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1.  Introduction 


The  advancement  of  robot  capabilities  and  functionality  has  changed  the  way  in 
which  Soldiers  perform  many  of  their  operational  tasks.  The  various  unmanned  air 
(Valvanis  2008),  ground  (Zych  et  al.  2013),  and  submersible  vehicles  (Clegg  and 
Rodgers  2015)  currently  deployed  have  significantly  impacted  present-day  warfare. 
Although  many  of  these  systems  have  shown  to  be  beneficial  and  effective  for 
mission  success,  traditional  control  of  these  systems  is  through  teleoperation  (Chen 
et  al.  2007;  Lichiardopol  2007).  While  teleoperation  may  be  necessary  and 
appropriate  for  situations  that  may  otherwise  require  soldiers  to  be  exposed  to 
hazardous  or  life-threatening  situations,  it  is  not  recommended  for  dismounted 
operations  (Chen  et  al.  2007).  Hence,  autonomous  robots  provide  a  solution  that 
takes  advantage  of  current  robot  sensing  and  intelligence  while  reducing  the 
cognitive  demands  on  the  Soldier,  allowing  robots  to  maintain  awareness  of  the 
operational  environment  (Kott  et  al.  2015).  The  implementation  of  autonomous 
robots  within  human  teams  carries  with  it  concern  regarding  human-robot 
interaction  (HRI)  and,  more  specifically,  human-robot  communication. 

Moving  beyond  teleoperation,  military  HRI  has  focused  on  integrating  multimodal 
communication  (MMC)  methods  that  leverage  the  natural  ways  in  which 
human-human  interaction  takes  place  and  the  commonly  employed  functionality 
for  human-computer  interaction  (Abich  et  al.  2015).  In  a  general  sense,  MMC  is 
sending  and/or  receiving  information  through  multiple  sensory  systems  (e.g., 
seeing  text  information  that  is  also  presented  auditorily).  In  terms  of  benefits  for 
signal-communication  processing,  MMC  systems  are  robust,  flexible,  efficient, 
intuitive,  and  redundant  (Dumas  et  al.  2009;  Partan  and  Marler  1999).  While  many 
robot  systems  are  equipped  with  multimodal  interaction  capabilities  (Barber  et  al. 
2013;  Harris  and  Barber  2014),  the  impact  of  each  communication  type  on  the 
Soldier’s  ability  to  perform  task  critical  operations  is  not  well  known.  Therefore, 
systematic  evaluation  of  the  components  that  comprise  the  transactions  between 
humans  and  robots  and  the  way  in  which  information  is  conveyed  is  critical  prior 
to  the  deployment  of  any  system  to  the  field. 

There  were  2  major  goals  of  this  experiment.  The  first  was  to  investigate  the  effects 
on  performance  and  operator  perception  of  various  exogenous  orientation  design 
cues  associated  with  a  visual  display  in  a  multimodal  interface  to  facilitate  squad- 
level  communication  within  a  dismounted  Soldier-robot  team.  In  particular,  this 
goal  focused  on  determining  whether  the  elements  of  visually  displayed  robot 
reports  provided  adequate  information  about  the  situational  context  so  the  Soldier 
could  quickly  determine  the  best  course  of  action  the  robot  should  take  without 
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being  cognitively  overloaded.  The  second  goal  was  to  investigate  Soldiers’ 
preferences  when  it  came  to  status  updates  from  a  robot  teammate  (e.g.,  reporting 
frequency  and  format).  Specifically,  this  aspect  of  the  experiment  focused  on 
understanding  the  relationship  between  robot-reporting  preferences,  task 
performance,  and  situation  awareness  (SA)  with  a  Soldier  population. 

1.1  Visual  Displays  for  Dismounted  Soldier-Robot  Teams 

The  military  has  stated  its  interest  in  pursuing  mobile  technology  for  dismounted 
squads,  such  as  smartphones  or  tablets,  because  of  the  high  processing  capabilities 
now  available  in  smaller,  lighter  platforms.  In  2014,  the  Federal  Mobile  Computing 
Summit  held  by  the  Department  of  Defense  solicited  a  US-based  vendor  that  can 
supply  an  NSA-compliant  device  with  a  12-24  hour  battery  life,  compatibility  with 
multiple  commercial  networks  and  the  ability  to  integrate  with  a  planned  Defense 
Information  Systems  Agency  application  store...  (Mazmanian  2014).  The  Army’s 
Communications-Electronics  Research,  Development  and  Engineering  Center  has 
been  working  on  the  Edge-Enabled  System  with  the  mission  of  developing  a 
handheld  interface  for  the  future  warfighter  and  is  currently  field  testing  command, 
control,  communications,  computers,  intelligence,  surveillance,  and 
reconnaissance — commonly,  C4ISR — technology  (Jones-Bonbrest  2015;  McCall 
2010).  Additionally,  the  Defense  Advanced  Research  Projects  Agency’s  (DARPA) 
Squad  X  Core  Technologies  program  is  geared  toward  developing  new, 
lightweight,  energy-efficient  technology  to  support  tactical  advantages  for 
dismounted  infantry  squads  while  avoiding  negative  repercussions  such  as 
cognitive  or  physical  overloading  (DARPA  2015). 

Visual  displays  have  the  potential  to  provide  a  vast  amount  of  information  within  a 
relatively  small  space.  A  single  image  can  convey  complex  ideas.  In  a  field  study 
assessing  the  effects  of  different  information-presentation  types  on  task 
performance,  Soldiers  indicated  visual  displays  were  “easier  to  follow”  and 
information  was  “easier  to  recall.”  They  also  exhibited  better  SA  when  using  a 
visual  display  compared  to  an  auditory  display,  potentially  because  graphical 
representations  might  facilitate  chunking  of  information  into  manageable  sizes 
(Glumm  et  al.  1999).  Studies  have  shown  that  the  decay  of  information  in  visual 
working  memory  can  be  gradual.  Further,  the  capacity  for  storing  visual  images  is 
larger  compared  to  auditory  working  memory.  Therefore,  the  larger  working 
memory  storage,  in  addition  to  the  chunking  of  information,  might  explain  the 
Soldiers’  performance  and  preference  when  using  visual  displays  (Card  et  al.  1983; 
Zhang  and  Luck  2009). 
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For  visual  displays  to  be  effective,  they  must  orient  the  attention  of  the  user  (Posner 
et  al.  1980).  Visual  perception  is  driven  by  both  internal  (endogenous)  and  external 
(exogenous)  orienting  events.  Endogenous  orientation  refers  to  “purposeful 
allocation  of  attentional  resources”  (e.g.,  scanning  an  arrival  screen  at  an  airport) 
while  exogenous  orientation  refers  to  “reflexive,  automatic  responses”  such  as 
attention  captured  by  a  bright  blinking  light  (Mayer  et  al.  2004).  For  the  purposes 
here,  we  will  be  concentrating  on  exogenous  orienting.  To  support  exogenous 
orienting,  non-content  specific  features  should  be  implemented,  meaning  the  cues 
used  to  attract  attention  should  not  be  normally  present  in  the  environment  or  signal. 
These  are  referred  to  as  feature  singletons  (Mulckhuyse  and  Theeuwes  2010). 
Highlighting  would  be  considered  an  exogenous  orientation  event  that  provides 
visual  cues  to  attract  attention  and  support  detection  (Posner  1980).  The 
highlighting  of  navigational  routes  and  objects  in  an  environment  is  an  example  of 
exogenous  orienting  because  it  elicits  a  reflexive  and  automatic  response  without 
having  predetermined  biases  (Mayer  et  al.  2004).  There  are  many  factors  that 
increase  the  chances  of  capturing  visual  attention,  such  as  size,  intensity,  color,  and 
transformation  (Wickens  and  Hollands  2000),  but  the  elements  of  highlighting 
should  contain  non-content  features.  In  other  words,  the  graphics  (e.g.,  labels, 
shapes,  colors)  used  should  not  be  found  within  any  of  the  images  or  environments 
to  reduce  any  confusion  and  misunderstandings  that  could  lead  to  errors  (Salcedo 
et  al.  2014;  Wogalter  et  al.  2002).  The  colors  used  in  this  experiment  contrast  highly 
with  the  environment  while  not  being  accompanied  by  common  associations,  such 
as  the  colors  red  indicating  a  warning  or  stop,  green  indicating  good  or  go,  and 
yellow  indicating  caution  or  slow  down  (Wogalter  et  al.  2002).  The  application  of 
such  features  becomes  apparent  when  developing  mobile  applications  that 
inherently  have  limited  visual  space. 

Although  mobile  devices  may  be  an  HRI  and  communication  solution  for  the  future 
warfighter,  it  is  imperative  such  technology  not  impair  primary  task  performance; 
therefore,  particular  care  must  be  taken  when  designing  the  interface.  Chittaro 
(2006)  states  that  simply  scaling  down  visual  information  from  a  desktop  computer 
or  a  larger  display  to  fit  a  mobile  device  is  not  as  straightforward  as  it  may  seem 
because  of  the  many  limitations  that  exist  with  the  mobile  device  (e.g.,  limited 
space,  display  ratio  reduction,  reduced  processing  capabilities,  and  different 
interaction  or  input  techniques).  Further,  mobile  devices  are  more  likely  to  be  used 
in  various  physical  environments  (e.g.,  inside/outside,  day/night)  and  must  be 
robust  enough  to  display  information  in  all  conditions.  Even  more  of  an  issue  is 
safety  while  performing  other  activities  during  interaction  with  a  mobile  device — 
a  major  concern  because  of  the  demand  on  the  user’s  attention,  which  has  limited 
allocation  capacity  (Kahneman  1973).  These  factors  are  crucial  for  a  dismounted 
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Soldier  as  the  increased  attentional  and  cognitive  demands  related  to  operating  the 
mobile  device  could  lead  to  catastrophic  and  life-threatening  events. 


1.2  Information  Sharing  within  Dismounted  Soldier-Robot 
Teams 

To  operate  effectively  as  a  team,  Soldiers  should  communicate  with  one  another 
before,  during,  and  after  completion  of  a  mission.  Communication  among  team 
members  can  be  facilitated  through  information  sharing  (e.g.,  information 
regarding  the  environment,  task  progress,  or  new  directives);  this  is  necessary  for 
the  development  of  an  individual’s  SA  while  working  within  a  team  (Salas  et  al. 
1995).  In  comparing  the  current  state  of  the  art  in  communicating  within 
human-robot  teams  (i.e.,  continuous  video  feeds)  to  that  of  communicating  within 
human-human  teams  (i.e.,  periodic  communication  of  relevant  information),  it  can 
be  concluded  that  humans  and  robots  have  different  mental  models  when  it  comes 
to  engaging  in  information  sharing.  In  other  words,  humans  and  robots  each  use 
different  rules  when  deciding  how  often  and  what  information  should  be  shared 
among  team  members.  This  issue  is  less  prevalent  within  human-human  teams 
whereby  team  members  draw  from  shared  mental  models  that  inform  them  of  when 
information  should  be  “pushed”  to  their  team  members  (Johannesen  et  al.  1994; 
MacMillan  et  al.  2004).  Thus,  humans  are  often  aware  of  their  human  teammates’ 
informational  needs. 

In  contrast,  current  robots  are  unaware  of  the  informational  needs  of  their  Soldier 
team  members.  To  combat  this,  current  robots  are  designed  to  feed  information  to 
Soldiers  in  a  continuous  fashion  via  live  video  feeds  (Scholtz  2003).  This  method 
of  information  sharing  is  not  ideal  for  dismounted  operations  for  2  major  reasons. 
First,  it  is  taxing  on  the  human  visual  and  cognitive  processing  systems  (Burke  et 
al.  2004;  Casper  and  Murphy  2003;  Yanco  and  Drury  2007).  Second,  it  requires  the 
Soldier-operators  to  spend  most  of  their  time  “heads  down”  (i.e.,  looking 
downward  at  a  visual  display)  as  they  continuously  control  or  monitor  their  robot 
teammate  (RCTA  2014).  Thus,  this  form  of  human-robot  interaction  makes  it 
difficult  for  dismounted  Soldiers  to  perform  other  tasking  while  managing  the 
robot. 

Unlike  current  Soldier-robot  teams,  whereby  a  Soldier  is  responsible  for  direct 
control  of  a  robot  team  member,  the  vision  for  future  Soldier-robot  teams  is  one  in 
which  robots  autonomously  accomplish  tasks  and  contribute  to  team  performance 
(RCTA  2014).  While  this  autonomy  will  allow  Soldiers  to  perform  other  tasks,  it 
will  become  even  more  important  for  robots  to  proactively  push  information  (in  the 
form  of  updates)  to  Soldiers  in  anticipation  of  their  informational  needs.  What  is 
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unknown  at  this  point  is  how  often  Soldiers  expect  to  receive  these  updates  and 
how  they  would  like  the  shared  information  to  be  presented  to  them  (e.g.,  via  text, 
imagery,  or  auditorily). 

1.3  Experiment  Purpose 

The  first  part  of  this  study  was  robot  assistance  (RA)  and  focused  on  the 
unidirectional  communication  transaction  from  a  robot  to  a  dismounted  Soldier 
teammate  via  a  visual  display  during  a  surveillance  task.  Specifically,  this  explores 
the  effects  of  various  exogenous  orienting  visual  display  cues  within  robot¬ 
generated  assistance  request  on  participants’  task  performance,  perceived  mental 
workload,  and  usability  preference.  Primary  task  performance  was  assessed  as  the 
percentage  of  correctly  detected  targets  during  a  signal-detection  task;  that  is,  the 
threat  detection  (TD)  task  described  in  Section  2.3.1.  Task  performance  was  also 
measured  as  the  time  it  took  participants  to  respond  to  reports  (i.e.,  intervention 
response  time)  and  the  percentage  of  correct  choices  made  (i.e.,  intervention 
accuracy)  referring  to  the  content  of  the  RA  requests.  Two  types  of  RA  requests 
were  investigated:  navigational  route  and  building  selection.  These  requests  were 
selected  based  on  current  robot  capabilities  and  expected  future-use  cases  in  which 
a  robot  needs  to  address  situations  where  both  options  have  equal  probability  of 
being  correct  and  the  robot  has  not  yet  learned  how  to  decide  on  its  own  (Barber  et 
al.  2015).  The  mental  workload  of  Soldiers  was  assessed  using  a  subjective  measure 
that  captured  demands  imposed  on  the  participants  and  on  their  interaction  with  the 
visual  display.  This  assessment  helped  identify  the  sources  of  mental  workload 
associated  with  the  display  designs.  Usability  preference  was  also  captured  using  a 
subjective  measure  that  provided  a  composite  score  across  all  items  to  indicate  the 
participants’  preference  for  display  configuration.  Five  central  research  questions 
were  of  interest  for  the  RA  part  of  this  study: 

1)  Is  performance  accuracy  of  the  primary  detection  task  affected  by  the  type 
(navigation  or  building)  or  display  version  of  the  robot  request? 

2)  Is  there  a  difference  in  accuracy  of  correct  responses  to  robot  assistance 
requests  for  both  navigational  route  and  building  selection  depending  on  the 
type  of  exogenous  visual  cues  displayed? 

3)  Is  there  a  difference  in  response  time  to  visual  reports  for  both  navigational 
route  and  building  selection  depending  on  the  type  of  exogenous  visual  cues 
displayed? 

4)  How  is  workload  affected  by  the  type  of  exogenous  visual  cues  displayed 
for  both  navigational  route  and  building  selection? 
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5)  How  is  usability  preference  affected  by  the  type  of  exogenous  visual  cues 
displayed  for  both  navigational  route  and  building  selection? 

The  second  part  of  this  study  was  robot  reporting  (RR)  and  focused  on  investigating 
Soldier  preferences  in  receiving  status  updates  from  a  robot  teammate.  Given  that 
there  is  little  research  indicating  how  often  Soldiers  expect  to  receive  status  updates 
from  an  autonomous  robot  teammate  (i.e.,  how  often  they  expect  the  robot  to  push 
information  without  being  asked),  this  task  attempted  to  address  these  questions  by 
taking  a  slightly  different  approach  to  the  problem.  Participants  were  instructed  to 
perform  an  individual  task  (i.e.,  the  TD  task),  with  the  option  of  checking  in  on  a 
robot  teammate  as  often  as  they  preferred  (via  the  ability  to  request  either  text  or 
image  reports  from  the  robot).  This  strategy  allowed  for  an  empirical  examination 
of  Soldier  preferences  for  robot  reporting  (i.e.,  number  and  type  of  reports)  and  the 
factors  that  may  affect  their  preferences  (e.g.,  primary  task  difficulty).  Four  central 
research  questions  were  of  interest  for  the  RR  part  of  the  study: 

1)  How  often  do  Soldiers  expect  to  receive  status  reports  from  a  robot 
teammate,  and  is  this  preference  influenced  by  the  difficulty  of  a  Soldier’s 
primary  task? 

2)  Is  there  a  general  preference  among  Soldiers  for  text-based  or  image -based 
reports  when  seeking  updates  from  a  robot  teammate? 

3)  How  is  SA  affected  by  the  number  and  type  of  robot  reports  requested  by 
the  Soldier? 

4)  Is  there  a  performance  tradeoff  associated  with  robot  reports,  whereby 
primary  task  performance  decreases  as  more  robot  reports  are  received  by 
the  Soldier? 

2.  Method 


2.1  Participants 

Twenty-nine  Soldiers  from  Fort  Benning,  Georgia’s  Officer  Candidate  School,  22 
males  and  7  females  (mean  age  =  26.8,  standard  deviation  [ SD ]  =  3.3),  volunteered 
for  and  completed  the  experiment.  Participation  was  voluntary,  and  no 
compensation  was  awarded.  Participants  all  had  normal  or  corrected-to-normal 
(e.g.,  glasses,  contact  lenses)  vision  and  were  screened  for  color-vision  deficiency 
(2  males  were  red/green  color  deficient  but  were  still  included  in  the  analysis  since 
they  were  not  classified  as  data  outliers).  None  of  the  participants  had  prior 
experience  with  the  simulator  or  multimodal  interface.  Based  on  responses  gathered 
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from  the  demographics  questionnaire,  participants  indicated  they  had  little 
knowledge  of  basic  robotics  technology  or  experience  interacting  with  robots. 


Figure  1  illustrates  the  points  during  each  scenario  when  the  simulation  was  paused 
and  measures  were  administered;  in  addition  to  situation-awareness  probes,  the 
NASA  Task  Load  Index  (NASA-TLX),  System  Usability  Scale  (SUS),  and 
Situation  Awareness  Rating  Technique  (SART)  were  used.  For  the  purposes  of  this 
study,  we  defined  a  route  as  the  path  traveled  between  2  street  corners.  Figure  1 
shows  most  of  the  measures  were  administered  at  the  end  of  each  route  (i.e.,  once 
the  Soldier  and  robot  reached  a  corner).  During  the  RR  scenarios  (i.e.,  RR1  and 
RR2),  however,  the  simulation  randomly  paused  twice  during  each  route  to 
administer  SA  probes. 


RA  Scenario 


1 .  NASA-TLX 

2.  SUS 

1 .  NASA-TLX 

2.  SUS 
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1 .  NASA-TLX 

1 .  NASA-TLX 

2.  SUS 

2.  SUS 

RR1  &  RR2  Scenarios 


* SA-1  probes  were  randomly  administered  twice 
during  each  RR  route  (8  times  per  RR  scenario). 


Fig.  1  Conceptual  illustration  depicting  points  at  which  the  simulation  was  paused  for  each 
scenario  and  one  or  more  measures  administered 

2.2  Questionnaires 


2.2.1  ishihara's  Test  for  Color  Deficiency 

This  color  deficiency  test  assessed  for  red  and  green  reduced  sensitivity,  which  is 
the  most  common  form  of  color-vision  deficiency  (Ishihara  2014).  Participants 
viewed  11  colored  plates  requiring  them  to  identify  numbers  or  trace  a  path.  If 
participants  answered  5  or  more  incorrectly,  they  were  considered  color  deficient. 
Participants  were  not  removed  if  they  were  assessed  to  have  a  color-vision 
deficiency,  but  it  was  documented  to  account  for  any  individual  differences. 

2.2.2.  Demographics  Questionnaire 

This  questionnaire  (shown  in  Appendix  A)  was  developed  in-house  and  gathered 
background  information  on  each  participant’s  age,  gender,  visual  acuity,  and 
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academic  level  of  achievement  (and  type  of  degree,  if  applicable).  Information  on 
military  experience — years  in  the  service,  rank,  military  occupational  specialty, 
number  of  deployments,  and  mission-related  experience — was  collected.  The 
questionnaire  asked  about  computer  use:  length  of  time  participants  have  been 
using  a  computer  and  how  often  as  well  as  the  ways  they  interact  with  the  computer; 
further,  video-game  use,  such  as  how  often  and  what  types,  as  well  as  any 
experience  working  with  them  was  recorded.  Finally,  experience  with  and 
knowledge  of  robotics,  both  military  and  commercial,  was  recorded. 

2.2.3  Cube  Comparison  Test 

This  spatial  orientation  test  (shown  in  Appendix  B)  assesses  the  ability  to  mentally 
rotate  and  compare  objects  in  space.  The  test  comprises  2  parts  and  each  part 
consists  of  21  items,  but  only  Part  1  was  used.  This  test  requires  participants  to 
compare  2  cubes  and  determine  whether  one  cube  can  be  rotated  to  match  the  other 
cube.  Participants  have  3  minutes  to  answer  as  many  items  as  possible.  A 
participant’s  score  is  the  number  of  correct  responses  minus  incorrect  responses; 
therefore,  guessing  is  not  encouraged. 

2.2.4  NASA-TLX 

The  NASA-TLX  (Hart  and  Staveland  1988)  is  a  perceived-workload  assessment 
comprising  6  subscales:  mental  demand,  physical  demand,  temporal  demand, 
effort,  frustration,  and  performance.  A  global  perceived-workload  measure  is  also 
calculated  by  averaging  the  6  subscales.  Each  subscale  was  scored  on  a  100-point 
scale.  For  the  RA  scenario,  the  unweighted  TLX  (shown  in  Appendix  C)  was 
administered  by  computer  at  the  end  of  each  route  (i.e.,  4  times).  For  the  RR 
scenarios,  the  unweighted  TLX  was  administered  after  2  routes  had  been  completed 
(i.e.,  2  times:  halfway  through  each  RR  scenario  and  at  the  end  of  each  RR 
scenario). 

2.2.5  SUS 

This  10-item  questionnaire  focused  on  perceived  usability  of  the  system;  that  is, 
hardware,  software,  and  equipment  (Brooke  1996).  Ratings  were  indicated  using 
5-point  Likert  items  (1  =  strongly  disagree;  5  =  strongly  agree).  The  composite 
score  represents  the  overall  usability  of  the  system  and  ranges  from  0  to  100  with 
higher  scores  indicating  higher  usability.  Specifically,  these  questions  focused  on 
the  participants’  interaction  with  the  device  during  the  RA  scenario.  The 
questionnaire  (shown  in  Appendix  D)  was  administered  at  the  end  of  each  route  of 
the  RA  scenario. 
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2.2.6  Free  Response  Questionnaire 

This  6-item,  open-ended  questionnaire  (shown  in  Appendix  E)  covered  positive  and 
negative  aspects  of  the  participants’  interaction  with  the  simulated  multimodal 
interface  device  and  their  preference  for  display  design,  and  asked  them  to  suggest 
any  improvements.  Specifically,  these  questions  were  focused  on  the  interaction 
with  the  device  during  the  RA  scenario.  The  questionnaire  was  presented  to 
participants  after  completing  the  RA  scenario. 

2.2.7  Situation  Awareness  Global  Assessment  Technique  (SAGAT) 

Probes 

During  RR  scenarios  RR1  and  RR2  a  modification  of  the  SAGAT  (Endsley  1988) 
was  used  to  assess  objectively  the  participants’  SA  at  Levels  1  (Perception)  and  2 
(Comprehension)  of  Endsley’s  SA  model  (Endsley  1995).  The  SAGAT  method 
pauses  a  simulation  to  ask  participants  questions  about  the  simulation  environment 
and  task(s)  being  performed.  For  this  experiment,  SAGAT  probes  (shown  in 
Appendix  F)  were  presented  to  participants  as  they  completed  each  RR  scenario. 
Each  time  the  simulation  paused,  participants  received  2  questions:  one  about  the 
safety  of  the  Soldier’s  environment  and  one  about  the  safety  of  the  robot’s 
environment.  (Section  2. 3. 3. 2  details  the  safety  rules  relevant  to  answering  each 
question.)  Specifically,  there  were  2  probes  presented  to  participants:  SA-1  and 
SA-2. 

SA-1  probes  appeared  at  random,  as  the  Soldier  moved  along  each  route.  These 
probes  assessed  Level- 1  SA  because  correctly  answering  the  questions  only 
required  the  participant  to  have  perceived  elements  within  the  environment  (i.e., 
higher-level  comprehension  of  said  elements  was  not  necessary).  The  same  2 
questions  were  presented  each  time  an  SA-1  probe  appeared.  The  first  question 
assessed  participant  SA  regarding  the  Soldier’s  environment  and  the  second 
question  assessed  participant  SA  regarding  the  robot’s  environment. 

In  contrast,  SA-2  probes  appeared  at  the  end  of  a  route  (i.e.,  once  a  comer  was 
reached).  These  probes  assessed  Level-2  SA  because  correctly  answering  the 
questions  required  the  participant  to  perceive  and  comprehend  elements  within  the 
environment  (i.e.,  a  conclusion  had  to  be  formed  based  on  information  obtained 
while  traversing  the  previous  route).  The  same  2  questions  were  presented  each 
time  an  SA-2  probe  appeared.  Similar  to  the  S A- 1  probes,  the  first  question  assessed 
participant  SA  regarding  the  Soldier’s  environment  and  the  second  question 
assessed  participant  SA  regarding  the  robot’s  environment. 

These  probes  served  as  an  objective  measure  of  SA  because  each  question  had  only 
one  correct  answer.  For  each  RR  scenario,  each  participant  answered  8  SA-1  probes 
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(2  per  route)  and  4  SA-2  probes  (one  at  the  end  of  each  route).  For  the  purposes  of 
this  experiment,  scores  on  the  SA-1  and  SA-2  probes  were  calculated  as  the 
percentage  of  correctly  answered  probes. 

2.2.8  Adapted  SART 

During  the  RR  scenarios,  a  modification  of  the  original  Situation  Awareness  Rating 
Technique  (Taylor  1990)  was  used  to  assess  participants’  subjective  perceptions  of 
their  own  SA  during  the  RR  scenarios.  The  original  SART  consists  of  10  items  that 
are  divided  into  3  subscales:  demand  on  attentional  resources,  supply  of  attentional 
resources,  and  understanding  of  the  situation.  Participants  responded  to  the  SART 
by  rating  their  level  of  agreement  with  each  item  using  a  7-point  Likert  scale 
ranging  from  1  (low)  to  7  (high).  The  SART  adapted  for  this  experiment  (shown  in 
Appendix  G)  contained  the  original  10  items  as  well  as  3  additional  items  specific 
to  the  participants’  knowledge  of  their  robot  teammate.  (These  latter  3  items 
pertaining  to  the  robot,  however,  are  not  included  in  the  results  of  this  report  as  they 
were  for  exploratory  analytical  use.)  Scoring  of  the  original  10  items  followed  the 
procedure  used  by  Endsley  et  al.  (2000).  The  mean  of  each  subscale  was  calculated 
and  then  the  mean  of  each  subscale  was  used  in  the  following  formula: 

(Understanding  of  the  situation)  +  (Demand  on  attentional 
resources)  -  (Supply  of  attentional  resources)  =  Overall  SART 
rating 

Higher  overall  SART  ratings  (based  on  the  original  10  items)  correspond  with 
higher  subjective  perceptions  of  one’s  own  SA;  lower  ratings  correspond  with 
lower  subjective  perceptions  of  one’s  own  SA. 

2.3  Experimental  Tasks 

A  general  overview  of  the  experimental  tasks  are  provided  first  and  then  followed 
by  a  detailed  description  of  each.  The  experimental  scenarios  were  composed  of  3 
tasks — TD,  RA,  and  RR — all  performed  within  the  Mixed  Initiative  experimental 
test  bed  (MIX)  (Barber  et  al.  2008;  Reinerman-Jones  et  al.  2010).  Given  that  this 
experiment  had  2  focuses,  there  were  2  types  of  scenarios:  RA  and  RR.  Both 
included  a  TD  task.  All  scenarios  simulated  a  reconnaissance  and  surveillance  task 
where  the  Soldier  and  robot  were  traveling  along  separate,  non-overlapping  routes 
within  the  same  portion  of  a  city.  Figure  2  is  a  conceptual  illustration  of  these 
routes:  the  left  box  represents  the  Soldier’s  route  and  the  right  box  represents  the 
robot’s  route.  Each  “side”  of  the  box  (left,  top,  right,  bottom)  represents  a  different 
route  through  the  city  (i.e.,  there  are  4  routes  traveled  during  each  scenario).  In  each 
scenario,  the  Soldier  and  robot  both  start  at  the  same  comer  of  their  respective 
portion  of  the  city  (bottom  left,  top  left,  top  right,  or  bottom  right)  and  travel  the 
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same  direction  (clockwise  or  counterclockwise).  Both  the  Soldier  and  robot  travel 
at  the  same  speed;  thus,  each  reaches  the  beginning/end  of  each  route  at  the  same 
time.  For  example,  if  the  robot  and  Soldier  start  at  the  bottom-left  comer  and  are 
traveling  clockwise,  then  they  will  reach  the  end  of  each  route  (i.e.,  comer)  at  the 
same  time. 


I  .  ‘ 


Fig.  2  Example  overhead  illustration  of  routes  traveled  by  the  Soldier  and  the  robot  during 
each  scenario;  initial  starting  location  and  travel  direction  were  randomized  for  each 
participant  but  the  Soldier  and  robot  always  started  in  the  same  relative  location 

In  the  RA  scenario,  the  participant’s  role  was  to  identify  potential  threats  in  their 
own  environment  and  periodically  respond  to  RA  queries  displayed  on  a  visual 
prompt  (Fig.  3)  representing  a  virtual  version  of  a  multimodal  interface  (MMI). 
Figure  4  shows  a  conceptual  mockup  illustrating  how  the  interaction  between  the 
Soldier  and  robot  took  place  through  the  MMI  during  this  scenario.  As  the  robot 
travels  through  the  environment  and  comes  across  an  uncertain  situation,  it  sends  a 
report  to  a  Soldier  via  the  MMI.  The  Soldier  responds  to  the  report  and  that 
information  gets  transmitted  back  to  the  robot  through  the  MMI. 

nn  IT-1, 


Fig.  3  Image  represents  the  virtual  MMI  as  a  prompt  on  the  screen  within  the  MIX 
environment 
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Fig.  4  Representation  of  the  RA-scenario  context;  the  MMI  provided  a  visual  interface  for 
the  robot  to  send  RA  queries  and  for  the  Soldier  to  respond,  creating  closed-loop 
communication 


In  the  RR1  and  RR2  scenarios,  one  of  the  participant’s  roles  was  to  aid  in  the 
development  of  a  robot’s  reporting  capabilities  by  showing  the  robot  how  often 
(and  in  which  presentation  format)  they  preferred  to  receive  information  from  the 
robot.  In  addition,  participants  were  told  they  also  were  helping  the  robot  learn 
which  routes  through  a  simulated  urban  area  were  safe  for  travel.  For  this  scenario, 
status  updates  (i.e.,  robot  reports)  from  the  robot  were  displayed  at  the  top  left  of 
the  participants’  display  (Fig.  5).  This  scenario,  thus,  introduced  a  different  use  for 
the  MMI  (i.e.,  to  request  information  from  the  robot). 


Fig.  5  Image  represents  an  alternative  use  of  the  virtual  MMI,  whereby  status  updates  can 
be  requested  from  the  robot 

2.3.1  TD  Task 

The  MIX  test  bed  was  customized  to  represent  the  first-person  perspective  of  a 
Soldier  traveling  through  a  generic  Middle  Eastern  urban  environment  (Fig.  6).  The 
Soldier’s  route  was  preplanned  and  did  not  require  the  participants  to  control  the 
Soldier’s  movement.  The  participants’  role  was  to  identify  potential  threats  in  the 
environment  (i.e.,  a  signal-detection  task)  by  capturing  photos  to  help  populate  a 
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robot  teammate’s  database  with  examples  so  it  could  more  effectively  carry  out 
mission  tasks  autonomously.  There  were  4  categories  of  characters  (i.e.,  events) 
within  the  environment:  friendly  Soldiers,  friendly  civilians,  enemy  soldiers,  and 
insurgents  (Fig.  7).  Each  category  included  at  least  5  different  types  of  characters. 
Enemy  Soldiers  and  insurgents  were  classified  as  threats  (i.e.,  signals),  and  an  equal 
number  of  each  was  presented.  An  equal  number  of  each  category  of  nonthreats 
was  also  present.  All  threats  were  identified  by  left-clicking  with  a  mouse  on  the 
character  within  the  environment.  No  feedback  was  provided  regarding  the 
accuracy  of  detection,  but  participants  did  hear  the  sound  of  a  camera  shutter  to 
indicate  they  were  capturing  photos. 


Fig.  6  Screenshot  of  the  MIX  test  bed  representing  the  first-person  perspective  of  the 
Soldier  traveling  through  a  Middle  Eastern  urban  environment 


Fig.  7  Examples  of  characters  used  within  the  TD  task  are  (1  to  r)  friendly  Soldier,  friendly 
civilian,  enemy  soldier,  and  insurgent  (armed  civilian) 

The  event  rate  of  the  TD  task — speed  at  which  threats  and  nonthreats  were 
presented  within  the  Soldier’s  environment — differed  among  the  RA  and  RR 
scenarios.  In  the  RA  scenario,  the  event  rate  was  presented  at  30/min  with  a  signal 
probability  of  13.33%  based  on  previous  research  (Abich  et  al.  2013).  In  the  RR1 
and  RR2  scenarios,  the  event  rate  of  the  TD  task  varied.  Across  the  2  RR  scenarios, 


Approved  for  public  release;  distribution  is  unlimited. 


13 


3  event  rates  were  encountered  by  each  participant:  low,  moderate,  and  high.  In  all 
3  event-rate  conditions,  signal  probability  (i.e.,  the  ratio  of  threats  to  nonthreats 
among  the  characters)  was  consistent  at  13.33%;  however,  the  number  of  events 
encountered  in  each  condition  differed.  The  event  rate  was  15/min  in  the  low  event- 
rate  condition,  30/min  in  the  moderate  condition,  and  60/min  in  the  high  condition. 
Figure  8  depicts  how  the  3  event-rate  conditions  were  distributed  across  the  RA, 
RR1,  and  RR2  scenarios.  The  RA  and  RR1  scenarios  both  had  a  constant,  moderate 
event  rate  while  the  event  rate  in  the  RR2  scenario  switched  from  either  low  to  high 
or  high  to  low  (depending  upon  the  assigned  counterbalancing  condition). 

Performance  on  the  TD  task  was  assessed  as  the  overall  percentage  of  correct 
responses.  False-positive  rates  were  very  low;  therefore,  signal-detection  theory 
indices  were  not  calculated. 

RA  Scenario  RR1  Scenario  RR2  Scenario 

: . □ . v  .* . H . . \ 


H  up  H  [p  CD  iyi 


Fig.  8  Conceptual  representation  of  how  the  event  rates  (low  =  15/min,  moderate  =  30/min, 

high  =  60/min)  differed  across  the  3  scenarios 

2.3.2  RATask 

While  participants  were  performing  the  TD  task  (i.e.,  the  primary  task),  visual 
prompts  randomly  appeared  and  required  participants  to  respond  to  RA  queries 
generated  by  a  robot  teammate  (i.e.,  the  secondary  task).  The  visual  prompts  were 
a  virtual  representation  of  the  MMI  (shown  in  Fig.  2).  The  robot  teammate  was 
never  viewed  by  participants,  as  the  robot  was  traveling  a  separate  route.  At  times, 
the  robot  required  assistance  because  it  could  not  deduce  the  best  option  based  on 
its  intelligence  capabilities.  The  types  of  assistance  the  robot  requested  were  for  1) 
navigational  routes  and  2)  building  identification  (Figs.  9  and  10).  The 
navigational-assistance  requests  asked  participants  to  decide  the  best  route  for  the 
robot  to  avoid  obstacles.  The  building-identification  assistance  requests  asked 
participants  which  building  a  robot  should  screen  (i.e.,  monitor).  The  information 
the  participants  needed  to  make  their  decision  was  gathered  from  the  MMI.  Every 
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MMI  prompt  comprised  2  images:  1)  the  right  image  represented  the  point  of  view 
(POV)  of  the  robot  traveling  through  the  environment  and  2)  the  left  image 
represented  an  aerial  view  of  the  operational  area  (Fig.  3).  Both  images  represented 
the  same  scene  but  from  different  angles;  therefore,  the  RA  queries  could  be 
answered  by  gathering  the  pertinent  information  from  either  image.  The  research 
interest  here  lies  in  evaluating  the  workload  impact  and  preference  of  visual  display 
format  by  comparing  the  types  of  display  formats  for  each  query  (i.e.,  Navigation 
A  vs.  B  and  Building  A  vs.  B).  Participants  responded  to  the  robot’s  query  by  left- 
clicking  on  one  of  the  buttons  located  at  the  top  of  the  screen  to  indicate  their  choice 
(shown  in  Fig.  3).  No  feedback  was  provided  to  the  participants  regarding  the 
accuracy  of  their  choices.  The  simulation  paused — Soldier’s  movement  through  the 
environment  stopped — when  each  RA  query  appeared  and  did  not  resume  until  a 
response  to  the  robot’s  request  was  made. 


Which  direction  should  l  navijate  around  this  round  obstacle,  route  A  or  B?  Wh«h  direction  should  I  navijate  around  this  round  obstacle,  route  A  or  B? 


Fig.  9  Two  images  representing  the  robot’s  navigational-assistance-query  display  formats 
in  the  MMI;  image  on  the  left  has  extended  directional  arrows  (Navigation  A)  while  the  image 
on  the  right  has  short,  bold  directional  arrows  (Navigation  B) 


Fig.  10  Two  images  representing  the  robot’s  building-identification-query  display  formats 
in  the  MMI;  image  on  the  left  has  highlighted  boxes  around  the  buildings  (Building  A)  while 
the  image  on  the  right  has  highlighted  boxes  around  the  buildings  with  modifications  to  the 
scene  and  aerial  images  (Building  B) 


The  navigational-route  RA  queries  displayed  route  options  as  either  extended 
segmented  arrows  (i.e.,  Navigation  A  design)  or  short,  bold  directional  arrows  (i.e., 
Navigation  B  design).  The  building-identification  RA  queries  displayed 
highlighted  boxes  around  the  buildings  of  interest  (i.e.,  Building  A  design)  or  had 
highlighted  boxes  with  enhanced  modifications  to  the  images  such  as  grayed-out 
areas  in  the  POV  of  the  robot  image  and  detailed  aerial  information  (i.e.,  Building 
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B  design).  The  queries  for  each  type  of  display  format  (i.e.,  navigation  and  building) 
were  the  same;  the  only  difference  was  the  way  in  which  the  information  was 
portrayed  on  the  MMI.  The  elements  comprising  exogenous  orientation 
highlighting  were  chosen  based  on  their  noncontent  features.  The  labels,  shapes, 
and  colors  had  no  relation  or  similarity  to  elements  within  the  environment.  The 
red  (R),  green  (G),  and  blue  (B)  color  values  for  the  magenta  were  R:  255,  G:  51, 
B:  204  and  for  the  green  were  R:  0,  G:  255,  B:  204.  The  scenario  lasted  about  10 
min,  and  there  was  a  total  of  32  RA  reports  (i.e.,  8  of  each  type  of  display  format): 
one  presented  (on  average)  every  15-20  s.  Each  route  in  the  scenario  was  about  2.5 
min  and  comprised  one  type  of  display  format.  The  order  in  which  the  display 
formats  were  presented  for  each  route  was  randomized  using  a  random-number 
generator  to  determine  the  presentation  order.  At  the  end  of  each  route,  the 
workload  and  usability  questionnaires  were  presented  (depicted  in  Fig.  1).  The 
order  in  which  each  participant  navigated  around  the  scenario  was  counterbalanced 
and  randomized,  meaning  participants  were  randomly  assigned  starting  locations  at 
one  of  the  4  comers  of  the  scenario  and  either  traveled  clockwise  or 
counter-clockwise  (shown  in  Fig.  2). 

2.3.3  RR  Task 

Participants  were  informed  their  role  in  the  RR  scenarios  (in  addition  to  performing 
the  TD  task)  was  to  aid  in  the  development  of  a  robot’s  reporting  capabilities  and 
to  aid  the  robot  in  determining  which  navigational  routes  through  a  simulated  city 
were  safe  for  travel.  Similar  to  the  RA  task,  participants  were  informed  the  robot 
was  traveling  along  a  series  of  routes  throughout  a  portion  of  the  city  separate  from 
the  one  traveled  by  the  Soldier.  Participants  were  instructed  they  were  responsible 
for  helping  the  robot  a)  build  a  database  of  potential  threats  in  the  environment,  b) 
decide  how  often  it  should  send  status  reports  and  what  type  (image  or  text  reports), 
and  c)  decide  which  routes  through  the  city  were  safe  for  travel.  Participants  were 
informed  that  during  the  RR  scenarios  they  were  to  identify  threats  from  the 
first-person  Soldier  perspective  (i.e.,  perform  the  TD  task,  the  primary)  while 
simultaneously  performing  the  RR  task  (requesting  robot  reports  and  assessing  the 
safety  of  the  Soldier’s  and  robot’s  routes,  the  secondary). 

Thus,  both  the  RA  and  RR  scenarios  required  participants  to  perform  the  TD  task, 
but  differed  in  terms  of  the  interactions  taking  place  between  the  Soldier  and  robot. 
In  the  RA  scenario,  the  robot  pushed  information  to  the  Soldier  (without  prompting) 
and  the  Soldier  could  not  request  information  (i.e.,  reports)  from  the  robot.  In 
contrast,  in  the  RR  scenarios,  the  robot  only  sent  information  to  the  Soldier  if  it  was 
requested  (via  a  text  or  image  report). 
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2. 3. 3.1  Robot  Reports 

During  each  RR  scenario,  participants  were  told  they  could  request  reports  from  the 
robot  that  informed  them  of  the  robot’s  current  environment.  Participants  were  told 
that  due  to  bandwidth  limitations,  the  robot  could  only  send  new  reports  (i.e., 
containing  updated  information)  every  10  s.  Each  report  included  a  report  number 
that  appeared  above  the  requested  report  to  clearly  identify  it  as  unique  (containing 
new  information).  These  report  numbers  “reset”  at  the  beginning  of  each  route  (i.e., 
whenever  the  Soldier  and  robot  reached  a  corner).  This  made  it  clear  to  participants 
that  a)  each  scenario  consisted  of  4  distinct  routes  and  b)  the  SA-1  and  SA-2  probes 
should  be  answered  based  on  only  the  current  route  (not  the  previously  traveled 
routes). 

Participants  had  the  option  to  receive  these  reports  in  text  or  image  format  by 
clicking  on  one  of  2  buttons  in  the  top-left  portion  of  their  screen  (Fig.  11).  When 
an  image  report  was  requested,  participants  were  shown  an  image  of  what  was 
being  viewed  from  a  forward-facing  camera  mounted  on  the  robot  (Fig.  12).  The 
image  showed  bounding  boxes  around  both  threats  and  critical  threats  to  assist  the 
Soldier  in  assessing  the  number  of  threats  present  in  the  robot’s  immediate 
environment.  Participants  could  also  request  text  reports  from  the  robot.  These 
reports  displayed  the  same  information  about  the  robot’s  immediate  environment 
but  in  a  text  format.  Specifically,  the  text  reports  provided  the  Soldier  with  the  total 
number  of  threats,  critical  threats,  and  nonthreats  present  (Fig.  12).  The  TD  task  did 
not  stop — simulation  did  not  pause — when  a  robot  report  was  requested.  The 
decision  to  include  a  continuous  TD  task  was  made  to  assess  more  accurately  the 
effect  requesting  and  reviewing  robot  reports  had  on  the  primary  task.  In  addition, 
participants  could  not  view  both  an  image  and  text  report  simultaneously  (i.e.,  have 
both  displayed  at  the  same  time);  however,  they  could  request  one  right  after 
another  and  view  the  same  information  in  both  report  formats  if  they  wanted  to. 


Fig.  11  Zoomed-in  view  of  the  portion  of  the  screen  dedicated  to  the  RR  task,  with  text- 
report  button  on  the  left  and  image-report  button  on  the  right 
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Fig.  12  (1  to  r)  Example  of  an  image  report  containing  one  threat  and  4  nonthreats  and  the 

same  report  requested  in  text  format 

2.33.2  Safety  Rules 

In  the  RR  scenarios,  participants  were  also  responsible  for  assessing  the  safety  of 
their  and  the  robot’s  environments.  To  accomplish  this  task,  participants  used  a  set 
of  rules  to  determine  the  safety  status  of  a  particular  environment.  Participants 
needed  to  use  these  rules  to  correctly  answer  3  of  the  4  SA  probes  (i.e.,  one  SA-1 
probe  and  both  SA-2  probes). 

The  only  rule  participants  had  to  use  to  identify  whether  the  Soldier’s  route  was 
safe  was  as  follows:  “If  three  or  more  critical  threats  are  seen  along  your  route,  then 
the  route  is  unsafe.”  Critical  threats  differed  from  other  threats  in  that  they  were 
armed  with  a  rocket  launcher.  If  3  or  more  of  these  were  present  along  the  Soldier’s 
route,  the  route  was  classified  as  “Unsafe”.  Participants  used  this  rule  to  correctly 
answer  the  following  SA-2  probe:  “Was  the  Soldier’s  route  safe?” 

Participants  were  also  responsible  for  classifying  the  safety  of  the  robot’s 
environment.  There  were  3  rules  used  to  classify  the  robot’s  environment  as  “Safe” 
or  “Unsafe”.  Two  of  these  rules  applied  to  the  safety  of  the  robot’s  immediate 
environment  (i.e.,  what  is  displayed  within  the  robot’s  report)  while  the  third 
applied  to  the  safety  of  the  route  the  robot  was  traveling.  The  2  safety  rules  relevant 
to  the  robot’s  immediate  environment  were  as  follows: 

1)  If  the  robot  sends  a  report  (either  text  or  image)  that  includes  3  or  more 
threats,  the  robot’s  immediate  environment  is  unsafe. 

2)  If  the  robot  detects  a  critical  threat,  the  robot’s  immediate  environment  is 
unsafe. 

Figures  13  and  14  illustrate  these  2  safety  rules  in  application. 
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Fig.  13  Robot’s  report  shows  its  immediate  environment  is  unsafe;  (left)  in  image  report  4 
threats  are  bounded  in  yellow  boxes  while  (right)  text  report  counts  the  total  number  of  threats 


Fig.  14  Robot’s  report  shows  unsafe  immediate  environment;  (left)  in  image  report  a  critical 
threat  is  bounded  in  blue  box  while  (right)  the  text  report  counts  the  number  of  critical  threats 

Thus,  both  of  these  rules  were  needed  to  correctly  answer  the  following  SA-1 
probe:  “Based  on  the  currently  available  robot  report,  is  the  robot’s  immediate 
environment  safe?” 

The  safety  rule  relevant  to  the  robot’s  route  was,  “If  the  robot  sends  three 
consecutive  unsafe  reports  while  traveling  a  particular  route,  then  that  route  is 
unsafe.”  For  example,  if  a  participant  requested  reports  No.  5-7  and  all  3  were 
unsafe,  then  the  entire  route  the  robot  was  traveling  on  was  unsafe.  This  rule  applied 
even  when  the  3  reports  were  a  combination  of  text  and  image  (e.g.,  Report  5  was 
image,  Report  6  was  text,  and  Report  7  was  image).  Participants  used  this  rule  to 
correctly  answer  the  following  SA-2  probe:  “Was  the  robot’s  route  safe?” 

So  participants’  SA  would  not  be  influenced  by  the  need  to  remember  these  4  rules, 
they  were  given  a  “cheat  sheet”  to  use  during  the  RR  scenarios.  This  cheat  sheet 
depicted  all  4  safety  rules  both  graphically  and  in  text  (see  Appendix  H).  In 
addition,  participants  were  informed  each  scenario  consisted  of  4  distinct  routes — 
each  side  of  the  boxes  depicted  in  Figs.  1,  2,  and  8  represent  a  distinct  route — and 
they  should  keep  that  in  mind  when  considering  the  safety  of  a  particular  route  and 
answering  the  SA  questionnaires  (SA-1  probes,  SA-2  probes,  and  SART). 
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2.4  Apparatus 


The  simulation  was  presented  using  a  standard  desktop  computer  or  laptop  with 
equal  specifications  (3.2-GHz,  Intel  Core  i7  processor)  connected  to  a  22-inch 
(16:10  aspect  ratio)  monitor.  Responses  to  the  tasks  were  collected  using  the  left 
mouse  button  and  keyboard.  Participants  wore  headphones  to  reduce  any 
interference  between  them  while  viewing  the  narrated  training  slides  and  listening 
to  any  sounds  generated  from  the  simulation. 

2.5  Procedures 

When  participants  arrived,  they  were  instructed  to  first  read  the  informed  consent 
(see  Appendix  I).  Upon  consent,  participants  were  assessed  for  color-vision 
deficiency;  then,  they  completed  the  demographics  form  and  the  Cube  Comparison 
test.  Task  training  followed  completion  of  the  prestudy  questionnaires. 

2.5.1  Training 

A  30-min,  narrated  PowerPoint  presentation  was  used  to  support  consistency  of 
training  for  each  participant.  Training  was  accomplished  in  3  phases.  The  first 
phase  was  12  min,  during  which  participants  were  instructed  on  the  continuous  task 
(TD)  they  were  to  perform  and  how  to  respond  to  the  NASA-TLX.  They  were  then 
given  the  chance  to  practice  the  task  for  about  1  min  and  respond  to  the 
NASA-TLX. 

The  next  2  phases  of  training  continued  in  the  same  format.  The  second  phase  was 
6  min  and  explained  the  purpose  and  elements  of  the  4  different  RA-query  display 
formats.  Instructions  were  also  provided  on  how  to  respond  to  the  usability  and 
free -response  questionnaires.  Participants  then  practiced  responding  to  2  display 
formats  of  RA  queries;  this  illustrated  the  way  queries  would  be  presented  and 
ensured  the  Soldiers  understood  how  to  respond  to  them.  The  third  phase  was  12 
min  and  described  the  RR  task  and  its  associated  SA  questionnaires  and  probes. 
Participants  practiced  performing  the  task  for  about  1  min  and  responded  to  3  SA 
probes. 
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2.5.2  Experiment's  Design 

Following  the  training,  each  participant  completed  3  experimental  scenarios:  one 
RA  and  2  RR.  To  reduce  the  level  of  confusion  regarding  the  rules  for  the  RA  and 
RR  tasks,  the  RR  scenarios  were  always  paired  in  consecutive  order  (e.g.,  scenario 
orders  RA,  RR1,  RR2  or  RR2,  RR1,  RA).  This  resulted  in  4  total  scenario- 
presentation  orders: 

.  RA,  RR1,  RR2 

.  RA,  RR2,  RR1 

.  RR1,  RR2,  RA 

.  RR2,  RR1,  RA 


Therefore,  the  order  of  presentation  of  the  3  experimental  scenarios  was 
randomized  but  only  semicounterbalanced.  In  addition,  the  starting/ending  location 
(i.e.,  starting/ending  corner)  within  each  scenario  as  well  as  the  direction  traveled, 
clockwise  or  counterclockwise,  were  fully  counterbalanced  to  account  for  possible 
order  effects. 


During  the  study,  up  to  15  soldiers  participated  at  the  same  time.  Although  grouped 
together  in  the  facility  (Fig.  15),  participants  worked  independently  of  each  other 
and  wore  headphones  during  tasks. 


Fig.  15  The  Fort  Benning  laboratory  while  14  Soldiers  participate  in  the  experiment  at  the 
same  time  (no  identifiable  information  can  be  gathered  from  photograph) 
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3.  Results 


3.1  RA  Task 

Paired  sample  /-tests  were  used  to  assess  the  impact  of  various  RA-query  designs 
on  task  performance,  perceived  workload,  and  display  preference.  Cohen’s  d  effect 
sizes  using  the  conventional  scale  of  0.2,  0.5,  and  0.8  (small,  medium,  and  large, 
respectively)  are  also  reported  for  specific  comparisons  of  RA-query  designs. 
Bivariate  correlations  were  run  to  find  relationships  among  demographic 
characteristics,  visual-spatial  skills,  performance,  perceived  workload,  and 
usability  preference.  The  sample  size  for  all  analyses  was  n  =  29.  The  analyses  were 
used  to  answer  the  5  research  questions  related  to  the  robot-assistance  task,  stated 
previously: 

1)  Is  performance  accuracy  of  the  threat-detection  task  affected  by  the  type 
(navigation  or  building)  or  display  version  of  the  robot’s  request? 

2)  Is  there  a  difference  in  accuracy  of  correct  responses  to  RA  requests  for  both 
navigational  route  and  building  selection  depending  on  the  type  of 
exogenous  visual  cues  displayed? 

3)  Is  there  a  difference  in  reaction  time  to  respond  to  visual  reports  for  both 
navigational  route  and  building  selection  depending  on  the  type  of 
exogenous  visual  cues  displayed? 

4)  How  is  workload  affected  by  the  type  of  exogenous  visual  cues  displayed 
for  both  navigational  route  and  building  selection? 

5)  How  is  usability  preference  affected  by  the  type  of  exogenous  visual  cues 
displayed  for  both  navigational  route  and  building  selection? 

3.1.1  TD  Performance 

Two  paired  sample  /-tests  were  run  to  assess  the  effects  on  TD-task-performance 
accuracy  of  responding  to  the  different  building  and  navigation  RA-report  designs. 
Results  in  Table  1  show  no  significant  differences  were  found  (p  >  0.3  and  0.6, 
respectively). 
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Table  1  Results  of  /-tests  for  TD  task  performance  for  both  building  and  navigation  display 
designs  with  mean  (M),  standard  deviation  (SI)),  and  Cohen’s  d  statistical  value  (cl)  reported 


Threat  detection 
accuracy  (%  correct) 


M _ SI) _ M _ SI) 

Building  A  Building  B 

88.51  22.12  91.19  13.41 

Navigation  A _ Navigation  B 

88.51  1935  89.66  22.01 


Note:  probability  of  rejection  of  the  hypothesis  (p)  >  0.05 


/-test  d 


-0.96 


0.05 


-0.46  0.15 


3.1.2  RA  performance 

Two  paired  sample  /-tests  were  run  to  assess  the  effects  of  RA-report  designs  on 
RA-response  accuracy.  Statistically  significant  results  were  found  for  the  difference 
in  percentage  of  correct  responses  for  building-identification  and  navigational- 
route  selection  (Table  2).  Participants  made  significantly  more  correct  choices 
when  responding  to  the  Building-A  and  Navigation-A  design. 

Table  2  Results  of  Mests  for  robot  assistance  response  accuracy  for  both  building  and 
navigation  display  designs 


Correct  response 
accuracy  (%  correct) 


a  p  <  0.05 

b  p  <  0.01 


M _ SI) 

Building  A 

57.33  18T5 

Navigation  A 

71.98  15.54 


M _ SI) 

Building  B  /-test  d 

45.26  16.84  2.2T  069 

Navigation  B 

55.17  22.79  3.28b  0.86 


Two  paired  sample  /-tests  were  run  to  assess  the  effects  of  RA  report  designs  on 
response  time  for  building  identification  and  navigation  route  selection.  Results 
show  that  no  significant  differences  were  found  (Table  3). 

Table  3  Results  of  /-tests  for  RA  response  time  for  both  building  and  navigation  display 
designs 

M  SD  M  SD 

Building  A  Building  B  /-test  d 

4.45  1.98  3.80  1.86  L34  0.34 

Navigation  A _ Navigation  B 

5.10  2.72  4.23  1.85  1.73  0.36 

Note:  p  >  0.05 


Average  response 
time  (s) 
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3.1.3  Questionnaires 


3. 1.3.1  NASA-TLX 

Two  paired  sample  /-tests  were  run  to  assess  the  effects  of  RA  report  designs  on 
perceived  workload,  and  statistically  significant  differences  were  found  between 
report  designs  for  mental  and  global  demand.  The  Navigation-A  design  elicited 
higher  perceived  demand  on  both  nonweighted  subscales.  Results  are  reported  in 
Table  4. 

Table  4  Results  of  /-tests  for  NASA-TLX  ratings  for  both  building-  and  navigation-display 
designs 


M 

SD 

M 

SD 

d 

Building  A 

Building  B 

/-test 

Mental 

17.41 

22.14 

17.07 

18.20 

0.197 

0.02 

Physical 

7.93 

14.73 

8.79 

13.41 

-0.623 

0.06 

Temporal 

14.14 

17.12 

13.97 

16.60 

0.108 

0.01 

Effort 

16.03 

20.59 

14.31 

16.84 

0.935 

0.09 

Frustration 

11.90 

20.59 

10.00 

16.09 

0.938 

0.10 

Performance 

18.97 

27.40 

13.28 

20.58 

1.516 

0.23 

Global 

14.40 

15.50 

12.90 

14.36 

1.509 

0.10 

Navigation  A 

Navigation  B 

Mental 

20.86 

27.09 

13.28 

16.05 

2.491a 

0.34 

Physical 

9.14 

17.06 

6.72 

13.11 

1.545 

0.16 

Temporal 

18.45 

23.72 

13.79 

18.88 

1.831 

0.22 

Effort 

16.55 

23.11 

15.69 

20.73 

0.604 

0.04 

Frustration 

17.59 

25.45 

13.28 

20.50 

1.314 

0.19 

Performance 

15.86 

23.19 

16.72 

22.88 

-0.393 

0.04 

Global 

16.41 

19.17 

13.25 

15.29 

2.320a 

0.18 

a  p  <  0.05 

3. 1.3. 2  SUS 

A  series  of  /-tests  was  run  to  assess  the  effects  of  RA  report  design  on  perceived 
usability.  Results  show  a  significant  difference  between  building-identification 
designs,  but  not  for  navigation  design  (Table  5).  Participants  rated  the  Building  B 
higher  in  usability. 

Table  5  Results  of  /-tests  for  usability  for  both  building-  and  navigation-display  designs 


M 

SD 

M 

SD 

/-test 

d 

Building  A 

Building  B 

SUS  rating 

78.28 

18.86 

81.41 

15.58 

-2.05  la 

.18 

Navigation  A 

Navigation  B 

77.66 

18.87 

79.83 

17.10 

-1.167 

.12 

a  p  <  0.05 
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3. 1.3. 3  Free  Response 


Three  independent  raters  conducted  an  evaluation  of  the  data  for  common  themes 
across  participant  responses  to  each  item.  Raters  began  by  organizing  the  textual 
responses  into  common  ideas  and  then  comparing  the  frequency  with  which  certain 
themes  occurred  in  the  text  for  each  item.  Common  themes  that  were  mentioned  by 
at  least  3  respondents  for  each  item  were  retained.  The  3  assessments  were  then 
compared  for  overlapping  patterns  identified  by  all  3  raters.  Table  6  shows  the 
results  of  the  assessment.  Within  the  table  are  the  common  themes,  the  number  of 
participants  who  made  that  comment,  and  an  example  response. 

Table  6  Common  themes  among  participant  responses  to  the  Free  Response  Questionnaire 
for  RA-display  formats;  numbers  in  parentheses  next  to  each  theme  are  total  participants 
reporting  each  theme  (an  example  of  which  is  shown) 


Item/question 

Theme 

Example  responses 

Positive  aspects  of 
the  device 

a)  Ease  of  use  (12) 

b)  Multiple  views  (6) 

c)  Benefit  to  Soldier  (7) 

a)  The  device  was  really  simple  and  easy  to  use. 

b)  The  different  perspectives  provided  for  quick  analysis 
and  decision-making. 

c)  It  doesn’t  place  any  Soldiers  in  harm’s  way  by  allowing 
areas  to  be  viewed  without  danger  or  risk  involved. 

Negative  aspects 
of  the  device 

Diversion  (4) 

I  had  to  divert  my  attention  away  from  the  task  to  make 
decisions  for  the  robot. 

Navigational 

information 

a)  No  preference  for  layout  (9) 

b)  Preferred  aerial  view  (5) 

c)  Clear  and  sufficient  information 

(5) 

a)  The  device  did  a  very  good  job  providing  navigation.  I 
did  not  have  a  preference. 

b)  I  looked  at  both  but  preferred  the  overhead  view  because 

I  was  able  to  distinguish  the  area  better. 

c)  The  device  provided  clear  and  concise  information  to 
choose  the  best  way  for  the  robot  to  travel. 

Building 

identification 

a)  No  preference  for  layout  (10) 

b)  Preferred  aerial  view  (5) 

a)  I  liked  having  both  views,  both  were  beneficial. 

b)  I  think  the  aerial  view  was  the  easiest  way  to  gauge  how 
to  give  an  answer  to  the  presented  question. 

The  2  items  that  were  not  included  in  Table  6  probed  for  elaboration  on  the  first  4 
items  and  asked  for  suggested  improvements.  These  items  were  not  included 
because  the  responses  did  not  provide  any  unique,  additional  information  compared 
to  the  first  4  items.  Few  participants  suggested  any  improvements,  except  one 
participant  requested  a  zoom  feature  be  added  to  provide  a  wider  perspective  of  the 
environment. 


Approved  for  public  release;  distribution  is  unlimited. 


25 


3.1.4  Bivariate  Correlations 


Bivariate  correlations  were  run  to  find  relationships  among  demographic 
characteristics,  spatial  skills,  task  performance,  perceived  workload,  and  usability. 
To  clarify,  only  the  significant  variables  are  listed  in  Table  7  and  they  refer  to  how 
often  participants  use  a  computer,  how  much  experience  they  have  playing  video 
games,  time  when  responding  to  the  RA  reports,  global  TLX  associated  with 
responding  to  RA  reports,  and  RA  report-design  usability. 
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Table  7  Bivariate  correlation  matrix  for  statistically  significant  relationships  among  demographics,  task  performance,  global  workload,  and  usability 


1 

2 

3 

4 

5 

6 

7  8  9  10 

1 .  How  often  use  a  computer 

2.  Working  with  video  games 

0.363 

3.  RA — response  time — Navigation  A 

-0.151 

0.194 

4.  RA — response  time — Navigation  B 

-0.364 

-0.013 

5.  RA — response  time — Building  A 

-0.062 

-0.106 

6.  RA — response  time — Building  B 

-0.244 

0.105 

7.  NASA-TLX — global — Navigation  A 

-0.399a 

-0.286 

0.429a 

8.  NASA-TLX— global— Navigation  B 

-0.430 a 

-0.318 

0.425a 

9.  NASA-TLX— global— Building  A 

-0.5 13b 

-0.270 

0.493b 

10.  NASA-TLX— global— Building  B 

-0.465a 

-0.324 

0.180 

11.  SUS — Navigation  A 

0.45  la 

0.382a 

-0.177 

-0.715b 

12.  SUS — Navigation  B 

0.331 

0.492b 

-0.400a 

-0.695b 

13.  SUS— Building  A 

0.468a 

0.437a 

-0.546b 

-0.607b 

14.  SUS— Building  B 

0.456a 

0.434a 

-0.301 

-0.696b 

d  p  <  0.05 

b  p  <  0.01 


The  correlation  matrix  shows  that  the  more  participants  used  a  computer,  the  less 
perceived  workload  was  elicited  during  the  task;  also,  they  reported  a  higher 
usability  preference.  Video-game  experience  was  also  shown  to  have  a  positive 
relationship  with  usability  preference.  For  the  most  part,  longer  reaction  times  were 
associated  with  higher  perceived  workload,  but  only  reaction  time  to  RA  reports  of 
the  Building-A  and  Navigation-B  designs  were  negatively  associated  with  usability 
preference  at  a  statistically  significant  level.  Overall,  NASA-TLX  and  usability 
preferences  were  negatively  correlated. 

3.2  RR  Task 

One-way  repeated-measures  analyses  of  variance  (ANOVAs),  chi-square  tests,  and 
bivariate  correlations  were  used  to  analyze  the  RR  data  and  to  address  the  4  central 
research  questions  of  the  robot-reporting  task.  As  stated  previously,  our  research 
questions  were 

1)  How  often  do  Soldiers  expect  to  receive  status  reports  from  a  robot 
teammate  and  is  this  preference  influenced  by  the  difficulty  of  a  Soldier’s 
primary  task? 

2)  Is  there  a  general  preference  among  Soldiers  for  text-based  or  image-based 
reports  when  seeking  updates  from  a  robot  teammate? 

3)  How  is  SA  affected  by  the  number  and  type  of  robot  reports  requested  by 
the  Soldier? 

4)  Is  there  a  performance  tradeoff  associated  with  robot  reports,  whereby 
primary  task  performance  decreases  as  more  robot  reports  are  received  by 
the  Soldier? 

To  ensure  the  data  used  for  the  moderate  event-rate  condition  (i.e.,  RR1  scenario) 
were  comparable  to  data  used  for  the  low  and  high  event-rate  conditions  (i.e.,  RR2 
scenario),  only  the  measures  corresponding  to  the  first  half  of  the  RR1  scenario 
(i.e.,  the  first  2  sides  of  the  box)  were  used  for  analyses  involving  rate  comparisons. 
This  decision  was  made  because  participants  spent  twice  as  much  time  performing 
RR  tasks  in  moderate  than  they  did  in  low  or  high  (Fig.  8).  Thus,  it  was  decided 
only  those  measures  corresponding  to  a  participant’s  initial  experience  with  the 
moderate  condition  would  be  used  to  ensure  the  data  used  for  each  event-rate 
condition  were  comparable  (i.e.,  were  based  on  a  comparable  amount  of  time  spent 
within  each  condition). 

The  sample  size  for  all  analyses  was  n  =  29.  The  effect  sizes  for  the  main  effect  of 
each  repeated-measures  ANOVA  were  calculated  using  partial  eta-squared  (rj2)  and 
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the  effect  sizes  for  each  post  hoc  comparison  (i.e.,  paired  sample  t-test)  were 
calculated  using  Cohen’s  d. 


3.2.1  Global  Workload 

A  one-way  repeated-measures  ANOVA  was  conducted — with  event  rate  (low, 
moderate,  or  high)  as  the  independent  variable  and  global  workload  as  the 
dependent  variable — to  assess  whether  the  different  event-rate  conditions  resulted 
in  differences  in  participants’  perceived  workload  (i.e.,  global  workload)  while 
completing  the  RR  scenarios.  The  goal  of  this  analysis  was  to  identify  whether  the 
event-rate  manipulations  resulted  in  the  perception  of  a  more  (or  less)  difficult 
primary  task.  It  was  found  there  was  not  a  significant  main  effect  of  event  rate  on 
global  workload:  Wilks’  X  =  0.88,  F( 2,  27)  =  1.96,  p  =  0.161,  partial  ip  =  0.126. 

To  further  investigate  whether  the  event-rate  manipulation  resulted  in  a  more 
difficult  primary  task,  we  ran  a  one-way  repeated-measures  ANOVA  with  the  event 
rate  (low,  moderate,  or  high)  as  the  independent  variable  and  TD  performance  (% 
correct)  as  the  dependent  variable.  It  was  found  there  was  a  significant  main  effect 
of  event  rate  on  TD  performance:  Wilks’  X  =  0.71,  F( 2,  27)  =  5.44,  p  =  0. 01,  partial 
ip  =  0.287.  To  further  investigate  differences  in  TD  performance  among  the  3 
event-rate  conditions,  post  hoc  comparisons  were  conducted  using  paired- samples 
Mcsts  and  a  Bonferroni  correction  (p  <  0.0167  was  considered  significantly 
different).  The  post  hoc  tests  indicated  participants  had  significantly  worse  TD 
performance  when  the  event  rate  was  high — M  =  87.93,  SD  =  15.53 — than  when 
the  event  rate  was  moderate:  M  =  96.55,  SD  =  5.64;  f(28)  =  -3.11,  p  =  0.004, 
Cohen’s  d  =  -0.683.  There  were,  however,  no  significant  differences  in  TD 
performance  between  the  moderate  and  low — M  =  91.38,  SD  =  15.75;  t{ 28)  =  1.68, 
p  =  0.103,  Cohen’s  d  =  0.348 — or  the  high  and  low  conditions:  t( 28)  =  -1.80,  p  = 
0.083,  Cohen’s  d  =  -0.333.  Figure  16  depicts  these  relationships. 
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TD  Performance  (+/- 1  SD) 
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Fig.  16  Graph  depicting  TD  performance  across  event-rate  conditions  for  the  RR  scenarios 

3.2.2  Number  of  Reports 

The  total  number  of  reports  requested  by  each  participant  during  each  route  was 
calculated  to  identify  the  average  number  of  robot  reports  requested  during  each 
RR  route.  It  was  found  the  Soldiers  on  average  (averaging  across  all  8  RR  routes 
regardless  of  event  rate)  requested  a  robot  report  11.3  times  during  each  route  or 
once  every  12.39  s. 

The  same  procedure  was  used  for  each  event-rate  condition  (i.e.,  averaging  across 
both  routes  within  each  condition)  to  calculate  the  average  number  of  robot  reports 
requested  during  each  RR  route  for  a  given  event-rate  condition.  Then,  a  one-way 
repeated-measures  ANOVA  was  conducted — with  event  rate  (low,  moderate,  or 
high)  as  the  independent  variable  and  the  number  of  reports  as  the  dependent 
variable — to  assess  whether  the  number  of  reports  requested  in  each  condition  was 
significantly  different.  It  was  found  there  was  a  significant  main  effect  of  event  rate 
on  the  number  of  reports  requested:  Wilks’  X  =  0.55,  F( 2,  27)  =  10.98,  p  <  0.001, 
partial  >f  =  0.448.  To  further  investigate  differences  in  the  number  of  reports 
between  the  3  event-rate  conditions,  post  hoc  comparisons  were  made  using 
paired-samples  7-tests  and  a  Bonferroni  correction  (p  <  0.0167  was  considered 
significantly  different).  The  post  hoc  tests  indicated  participants  requested 
significantly  fewer  robot  reports  when  the  event  rate  was  high — M  =  9.19,  SD 
=  6.74 — than  when  it  was  moderate:  M=  12.90,  SD  =  9.51;  7(28)  =  -3.12 ,p  =  0.004, 
Cohen’s  d  =  -0.633.  Participants  also  requested  significantly  fewer  robot  reports 
when  the  event  rate  was  high  than  when  the  event  rate  was  low:  M  =  11.67,  SD  = 
7.27;  7(28)  =  -4.19,  p  <  0.001,  Cohen’s  d  =  -0.789.  There  was  no  significant 
difference  in  the  number  of  requested  reports  between  the  moderate  and  low 
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conditions:  t( 28)  =  1.01,  p  =  0.322,  Cohen’s  d  =  0.198.  Figure  17  depicts  these 
relationships. 

Number  of  Reports  (+/-  1  SD) 


Event  Rate  Condition 


Fig.  17  Graph  depicting  number  of  reports  across  event-rate  conditions  for  the  RR 
scenarios 


3.2.3  Report-Format  Preference 

To  identify  participants’  preference  when  it  came  to  report  format,  counts  were 
used  to  identify  the  total  number  of  text  and  image  reports  requested  by  each 
participant  across  the  2  RR  scenarios  (i.e.,  all  event -rate  conditions).  Participants 
who  requested  more  of  one  report  format  (>  50%  of  total  reports  requested)  than 
the  other  were  assumed  to  prefer  the  given  format.  For  example,  if  a  participant 
requested  80  text  reports  and  30  image  reports,  it  was  assumed  the  participant 
preferred  text  reports. 

In  total,  17  participants  preferred  the  text  reports  while  12  preferred  the  image 
reports.  A  chi-square  test  was  run  to  assess  whether  this  distribution  was  greater 
than  chance.  It  was  found  the  distribution  of  preferences  was  not  statistically  greater 
than  chance:  x2(l)  =  0.86,  p  >  0.05.  Furthermore,  it  was  found  that  2  of  the 
participants  exclusively  used  text  reports  (never  requested  an  image  report)  and  2 
of  the  participants  exclusively  used  image  reports  (never  requested  a  text  report). 
Finally,  it  was  found  that  22  participants  (75%)  were  consistent  in  their  preference 
for  either  text  or  image  reports,  requesting  more  of  their  preferred  format  than  the 
other  format  during  6  out  of  the  8  RR  routes.  An  additional  chi-square  test  was  run 
to  assess  whether  this  finding  was  greater  than  chance.  It  was  found  the  distribution 
was  statistically  greater  than  chance:  x2(l)  =  7.76,  p  =  0.005.  Close  inspection  of 
the  data  for  these  participants  indicated  report-format  preference  was  independent 
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of  the  event  rate  (i.e.,  participants  stayed  with  their  preferred  format  regardless  of 
the  event  rate). 

3.2.4  Bivariate  Correlations 

A  series  of  bivariate  correlations  was  run  to  examine  1)  the  relationship  between  3 
measures  of  SA  relevant  to  the  Soldier’s  environment:  SA-1  probe,  SA-2  probe, 
and  overall  SART  rating;  2)  participants’  RR  preferences  via  the  number  of  reports 
requested  and  report-format  preference  (text  preference  was  coded  as  0;  image 
preference  was  coded  as  1);  and  3)  primary  task  performance  (i.e.,  TD 
performance).  These  correlations  were  based  upon  the  averages  across  all  8  RR 
routes,  regardless  of  event-rate  condition.  Table  8  provides  the  correlations  among 
these  sets  of  variables.  There  was  a  significant  positive  correlation  between  the  total 
number  of  reports  requested  and  performance  on  the  SA-2  probes  (r  =  0.559,  p  = 
0.002).  Most  of  the  remaining  variable  pairs  were  positively  correlated  but  none  of 
these  correlations  reached  statistical  significance. 

Table  8  Bivariate  correlation  matrix  for  situation-awareness  measures,  reporting 
preference,  and  primary  task  performance 


1 

2 

3 

4  5  6 

1 .  Number  of  reports 

2.  Report-format  preference 

0.293 

3.  SA-1  probes — %  correct 

0.222 

0.059 

4.  SA-2  probes — %  correct 

0.559a 

0.233 

0.323 

5.  Overall  SART  rating 

0.117 

-0.101 

0.221 

0.220  ... 

6.  TD  performance — %  correct 

0.074 

0.122 

0.171 

0.242  -0.030  ... 

ap<0.01 


4.  Discussion 


4.1  Robot  Assistance 

The  first  goal  of  this  study  was  to  explore  the  use  and  effects  of  various  exogenous 
orientation  visual  cues  within  visual  reports  generated  by  a  robot  on  a  mobile 
platform  to  convey  squad-level  information  to  a  Soldier  teammate  in  a  dismounted 
scenario.  The  findings  support  expectations  that  design  differences  have  an  impact 
on  Solders’  perceived  mental  workload,  usability,  and  response  performance  to 
robot  reports.  Each  cue  design  type  (i.e.,  navigation  and  building)  is  discussed 
separately. 

4.1.1  Navigational  Routes 

For  TD-task  performance,  no  effect  was  shown  for  either  type  of  navigation  report. 
This  is  likely  due  to  the  TD  task  being  held  at  a  constant  rate.  Additionally,  when 
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participants  responded  to  the  RA  reports  the  simulation  paused,  which  meant  the 
cost  of  looking  at  the  MMI  on  TD  performance  was  not  assessed  during  this 
experiment. 

The  performance  for  the  RA  reports  shows  that  participants  responded  correctly 
more  often  when  viewing  the  Navigation-A  reports  id  =  0.86).  The  exogenous 
orienting  visual  cues  in  the  Navigation-A  design  displayed  options  that  showed 
participants  not  only  which  way  the  robot  could  go,  but  also  whether  the  robot 
would  encounter  any  obstacles  along  each  route  and  where  the  robot  would  end  up. 
The  additional  information  conveyed  in  the  Navigation-A  design,  compared  to  the 
Navigation-B  design,  seemed  to  support  decision  making  by  visually  displaying  the 
consequences  of  each  option.  Further,  responding  to  the  Navigation-A  reports 
elicited  higher  perceived  mental-demand  and  global-workload  ratings  than 
Navigation-B  reports,  which  could  be  partially  explained  by  the  difference  in  the 
amount  of  information  visually  displayed.  Interestingly,  mean  ratings  of  the 
Navigation-B  design  were  higher  in  terms  of  usability,  but  the  effect  was  fairly  low 
(d  =  0.12)  and  not  statistically  significant;  therefore,  based  on  this  sample,  no 
preference  was  indicated,  which  corresponds  to  the  open-ended  responses.  A  key 
point  is  response  accuracy  to  RA  reports  might  be  inherently  linked  with  perceived 
workload,  meaning  display  designs  might  need  to  elicit  a  certain  level  of  perceived 
demand  to  facilitate  decision  making  and  engagement,  potentially  through  more 
content-rich  visualization.  This  is  related  to  past  research  that  found  too  little  mental 
workload  (Abich  et  al.  2013)  or  too  much  could  lead  to  declines  in  task  performance 
(Hancock  and  Warm  1989). 

Taking  into  account  the  individual  differences,  it  seems  the  more  often  a  Soldier 
uses  a  computer,  the  lower  the  elicited  workload  response  (as  shown  in  Table  7), 
which  makes  sense  considering  the  entire  experiment  was  conducted  on  a  computer 
and  the  MMI  is  designed  on  a  computer-based  platform.  Overall,  these  findings 
show  that  even  though  task  performance  may  be  better  supported  by  a  specific 
visual-display  design,  if  there  is  an  increase  in  perceived  cognitive  demand  the 
usability  ratings  of  system  will  decline. 

4.1.2  Building  Identification 

Similar  to  the  navigation-display  designs,  no  differences  were  found  for  TD-task 
performance.  This  could  be  explained  by  the  same  interpretation  above. 
Additionally,  and  similar  to  the  Navigation-A  design,  results  indicate  responses  to 
Building-A  reports  had  better  accuracy  (d  =  0.69).  The  design  for  Building-A 
reports  simply  placed  highlighted  boxes  around  the  areas  of  interest  as  opposed  to 
desaturating  the  visual  information  outside  the  highlighted  areas  as  in  the  Building- 
B  design.  Therefore,  in  terms  of  display-format  design,  all  of  the  information 
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displayed  visually  for  object  identification  contributes  to  task  accuracy  and  should 
be  included  because  it  provides  contextual  information  that  assists  identification 
(De  Graef  et  al.  1990). 

When  looking  at  the  effects  on  perceived  workload,  although  not  statistically 
significant,  reported  mean  scores  show  the  Building-B  design  elicited  lower  ratings, 
which  might  have  contributed  to  the  significant  effect  on  usability  ( d  =  0.18);  yet, 
the  free-response  questionnaires  showed  no  stated  preference.  Further, 
response-accuracy  performance  was  also  lower.  This  could  be  explained  thus: 
having  too  low  of  a  mental  load  potentially  leads  to  disengagement  and  poor 
decision  making — as  discussed  previously  (Abich  et  al.  2013).  Further,  correlations 
showed  response  times  to  either  building  design  were  positively  associated  with 
NASA-TLX  ratings  and  negatively  related  to  usability  ratings  (Table  7),  suggesting 
both  designs  are  related  to  task  performance,  cognitive  demand,  and  display 
preference. 

4.1.3  Limitations  and  Future  Research 

When  the  participants  responded  to  the  RA  reports,  the  simulation  paused.  This  was 
a  known  limitation  implemented  intentionally  to  first  assess  the  quality  of  the  RA 
report  designs;  but,  to  gain  more  ecological  validity  it  would  be  best  to  have  the 
primary  task  (e.g.,  TD  task)  continue  as  it  would  in  the  real  world.  Manipulation  of 
this  factor  would  support  the  quantified  evaluation  of  the  cost  associated  with  MMI 
interaction  in  terms  of  task  performance,  cognitive  impact,  and  SA.  (A  subsequent 
study  is  planned  to  assess  this  impact  on  Soldiers  receiving  robot  reports  from 
multiple  robot  teammates  using  nonvisual  displays,  such  as  auditory  or 
visual-auditory  combination,  during  a  cordon- and- search  task.) 

Further,  the  extensive  control  of  the  laboratory  settings  allows  for  precise 
assessment,  but  limits  the  ecological  validity  of  the  findings  as  well.  Ultimately, 
the  results  from  this  program  of  study  will  drive  design  of  deployable  products  that 
will  support  Soldiers  in  a  dismounted  environment  and,  therefore,  a  balance  of  both 
field  and  lab  studies  is  necessary  to  foster  a  transfer  of  findings  from  the  lab  to  the 
field. 

The  intention  of  the  free-response  questions  was  to  probe  participants  about  their 
interactions  with  the  MMI  and  to  provide  any  suggestions  for  improvement. 
Although  some  participants  did  provide  feedback  regarding  the  MMI,  it  seems 
more  responses  were  directed  toward  the  capabilities  of  the  robot — even  though 
participants  were  told  that  evaluating  the  robot  was  not  the  focus  of  the  study.  In 
fact,  the  robot  was  never  seen — just  reports  from  the  robot  were  displayed — yet 
participants  were  still  evaluating  the  (notional)  robot. 


Approved  for  public  release;  distribution  is  unlimited. 


34 


The  next  step  will  be  to  explore  the  effects  of  robot  reports  conveyed  through  other 
modalities,  such  as  auditory  or  tactile,  on  dismounted  Soldier  performance  and  to 
use  a  more  dynamic  approach  (i.e.,  higher-fidelity  simulation). 

4.2  Robot  Reporting 

The  second  goal  of  this  study  was  to  explore  Soldier  preferences  when  it  came  to 
receiving  status  updates  (in  the  form  of  robot  reports)  from  a  robot  teammate.  Also 
of  interest  was  the  impact  these  reporting  preferences  had  on  participant  SA  and 
primary  task  performance. 

4.2.1  Task  Difficulty 

The  analysis  of  both  global  workload  and  TD  performance  served  the  purpose  of 
assessing  whether  the  event-rate  manipulation  resulted  in  a  more  or  less  difficult 
primary  task  for  participants.  We  expected  to  find  the  low  event-rate  condition 
would  be  the  least  difficult  (i.e.,  lowest  global-workload  scores  and  highest  TD 
performance),  high  to  be  the  most  difficult  (i.e.,  highest  global-workload  scores  and 
lowest  TD  performance),  and  moderate  to  be  between  low  and  high  in  difficulty  in 
terms  of  both  global  workload  and  TD  performance.  Instead,  our  results  were 
inconclusive  (as  shown  in  Figs.  16  and  17).  Participants  did  not  perceive  the  event- 
rate  manipulations  as  different,  from  a  workload  standpoint,  and  their  performance 
was  similar  across  most  of  the  conditions.  The  only  difference  found  was  in  TD 
performance  between  the  moderate  and  high  conditions. 

Overall,  it  appears  the  TD  task  may  have  been  a  bit  too  easy  for  participants. 
Evidence  of  this  comes  from  the  fact  that  event-rate  manipulation  did  not 
substantially  impact  participants  (from  either  a  workload  or  a  performance 
perspective).  Looking  at  the  means  for  the  3  conditions  for  both  global  workload 
and  TD  performance,  we  see  further  evidence  of  this  possible  explanation.  In  all  3 
conditions,  the  mean  TD  performance  (i.e.,  accuracy)  was  more  than  87%.  In 
addition,  mean  global  workload  was  below  30  (the  scale  is  0-100)  for  all  3 
conditions.  Taken  together,  this  indicates  participants  tended  to  perform  well  at  the 
TD  task  and  perceive  it  to  be  minimally  demanding  across  all  3  conditions. 

4.2.2  Reporting  Frequency 

It  was  anticipated  participants  would  request  fewer  reports  from  the  robot  as  the 
event  rate  increased  due  to  increased  demands  on  the  participant.  In  particular,  we 
believed  participants  would  spend  more  time  focused  on  the  TD  task  (due  to  the 
increased  number  of  threats  in  the  Soldier’s  environment)  with  less  time  spent 
requesting  reports  from  the  robot.  We  expected  a  linear  relationship  whereby  most 
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reports  would  be  requested  in  the  low  event-rate  condition,  the  second  most  in  the 
moderate  condition,  and  the  least  in  the  high  condition.  This  hypothesis  was  only 
partially  supported.  The  high  condition  did  result  in  fewer  reports  being  requested 
than  in  both  the  low  and  moderate  conditions,  but  a  similar  number  of  reports  were 
requested  in  both  low  and  moderate  (Fig.  17). 

Overall,  while  the  pattern  of  results  did  not  match  what  we  predicted,  there  were 
differences  among  the  conditions.  These  findings  highlight  the  importance  of 
considering  how  “busy”  one’s  team  member  may  be  when  deciding  how  often  to 
push  information  to  them.  Considering  that  Soldiers  must  perform  a  variety  of  tasks 
that  are  more  or  less  time/resource  intensive,  this  finding  provides  initial  evidence 
that  having  a  robot  send  reports  to  a  Soldier  teammate  based  on  a  strict  rule  (e.g., 
Once  every  X  seconds)  may  not  be  ideal.  Instead,  it  may  be  more  appropriate  to 
calibrate  sending  of  reports  based  on  nature  of  the  task  the  Soldier  will  be 
performing. 

4.2.3  Format  Preference 

When  it  came  to  Soldier  preferences  for  a  specific  report  format,  more  Soldiers 
preferred  text  reports  than  image  reports,  albeit  this  finding  was  not  significant  (i.e., 
this  observed  preference  could  be  due  to  chance).  While  the  Soldiers  were  not 
formally  asked  to  describe  why  they  chose/preferred  one  format  over  the  other, 
several  Soldiers  informally  shared  their  thoughts  during  (or  after  completing)  the 
experiment.  Soldiers  who  preferred  the  text  reports  mentioned  the  reports  helped 
them  process  the  information  they  needed  quicker  than  image  reports.  This  was 
because  the  text  reports  presented  participants  with  the  number  of  threats,  critical 
threats,  and  nonthreats  present,  whereas  the  image  reports  required  the  participants 
to  add  up  the  number  of  threats  and/or  critical  threats  seen  in  the  report.  As  for  the 
image  reports,  Soldiers  in  favor  of  these  reports  may  have  found  it  easier  to  keep 
track  of  the  reports  due  to  each  looking  distinctly  different  from  previous  reports. 
Some  Soldiers  mentioned  having  difficulty  quickly  identifying  the  number  of 
threats  and  critical  threats  with  the  text  reports  due  to  the  font  being  similar  for  each 
category  (i.e.,  same  font  style,  size,  and  color). 

Seventy-five  percent  of  the  participants  were  consistent  in  their  preference 
throughout  most  of  the  RR  scenarios;  that  is,  they  identified  a  preferred  reporting 
format  during  the  first  or  second  RR  route  and  stuck  to  that  preference  throughout 
the  remainder  of  the  RR  routes,  regardless  of  the  event  rate.  The  remaining 
participants  may  have  switched  back  and  forth  to  experiment  with  the  2  formats 
before  deciding  which  they  preferred.  The  fact  most  participants  immediately 
decided  upon  a  preferred  report  format  is  evidence  it  may  be  appropriate  to  allow 
Soldiers  to  inform  their  robot  teammate  of  their  preferred  reporting  format  before 
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beginning  a  mission.  Only  4  Soldiers  exclusively  requested  one  format  throughout 
all  RR  routes.  Thus,  25  out  of  29  (86%)  of  the  Soldiers  used  both  report  formats  in 
some  capacity  throughout  the  RR  scenarios.  This  finding  is  initial  evidence  that 
despite  having  a  general  preference  for  a  specific  reporting  format,  Soldiers  may 
still  like  to  have  another  format  available  to  them.  It  is  possible  the  Soldiers  may 
have  wanted  to  use  both  report  formats  to  double  check  the  information  in  one 
report  format  against  the  other.  Alternatively,  at  certain  times  during  their  tasking, 
having  both  formats  may  have  helped  them  feel  more  comfortable  in  understanding 
the  information  that  was  currently  available.  However,  because  the  Soldiers  were 
not  directly  asked  about  their  preference,  we  can  only  speculate  on  the  reasons  why 
both  formats  were  used  to  such  an  extent. 

4.2.4  Soldier  SA 

Despite  the  fact  most  linear  relationships  among  the  reporting  variables  (total 
number  of  reports  and  report  format  preference)  and  Soldier  SA  variables  (SA-1, 
SA-2,  and  SART)  were  not  strong  enough  to  reach  statistical  significance,  a 
discussion  of  the  data  trends  is  still  relevant.  In  particular,  it  was  expected  there 
would  be  a  negative  correlation  between  the  total  number  of  reports  and 
performance  on  both  the  Soldier  SA-1  and  SA-2  probes.  In  actuality,  these  2 
relationships  were  positive.  This  means  participants  who  requested  more  reports 
from  their  robot  teammate  tended  to  perform  better  when  asked  about  the  Soldier’s 
environment  within  the  MIX  simulation. 

This  finding  is  puzzling — it  would  seem  participants  who  spent  more  time 
reviewing  robot  reports  would  be  less  able  to  keep  track  of  the  number  of  critical 
threats  identified  along  the  Soldier’s  route.  One  possible  explanation  for  this 
finding  may  be  that  participants  who  requested  more  reports  were  more  engaged  in 
the  tasks  in  general.  In  other  words,  participants  who  were  requesting  reports  more 
frequently  may  have  been  more  motivated  to  perform  the  experimental  tasks  in 
general  and,  therefore,  kept  better  track  of  the  total  number  of  critical  threats  present 
along  the  Soldier’s  route. 

The  finding  that  TD  performance  was  positively  correlated  to  both  Soldier  SA 
probes  (as  opposed  to  negatively  correlated)  further  supports  this  possible 
explanation.  Another  interesting  finding  is  report-format  preference  was  positively 
correlated  with  both  the  SA-1  and  SA-2  probes  but  negatively  correlated  with  the 
SART.  This  provides  initial  evidence  that  participants  who  preferred  image  reports 
tended  to  perform  better  on  both  the  SA-1  and  SA-2  probes  (i.e.,  had  higher  SA) 
while  simultaneously  having  lower  perceived  SA  than  participants  who  preferred 
text  reports.  Taken  together,  these  relationships  indicates  that  participants  who 
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preferred  image  reports  may  have  believed  they  were  less  aware  of  the  Soldier’s 
environment  than  they  actually  were. 

4.2.5  Limitations  and  Future  Research 

Three  limitations  related  to  the  RR  task  should  be  considered. 

First,  despite  the  fact  the  event  rate  of  the  primary  task  was  manipulated  to  achieve 
low,  moderate,  and  high  task  difficulty  (based  on  previous  research  with  an 
undergraduate  student  population),  participants  did  not  necessarily  perceive  the 
different  event-rate  conditions  as  expected  (nor  did  they  perform  as  expected). 
Participants  may  not  have  felt  the  event-rate  manipulations  were  substantial  enough 
to  change  their  perceptions  of  the  primary  task.  Future  research  should  pilot-test 
any  difficulty/workload  manipulations  on  the  population  of  interest  before  full 
implementation  with  the  final  sample. 

A  second  limitation  related  to  the  RR  task  is  that  participant  preferences  for  robot- 
report  format  and  frequency  were  completely  derived  from  behavioral  data  (i.e., 
total  number  of  reports  and  the  number  of  each  type  of  report  requested).  While  the 
use  of  behavioral  data  resulted  in  interesting  insights,  the  inclusion  of  surveys  or 
questionnaires  would  have  allowed  participants  to  share  their  opinions  and 
elaborate  on  why  they  preferred  one  format  to  another.  Without  these  subjective 
data,  it  is  difficult  to  know  exactly  why  a  given  participant  chose  one  format  over 
another.  Future  research  could  address  this  by  including  a  pre-  and  post-task  survey 
that  asks  participants  for  their  opinions  of  the  2  different  reporting  formats. 

Finally,  future  research  should  also  consider  other  factors  that  may  influence  a 
Soldier’s  preference  for  robot  reporting  such  as  reliability,  trust,  and  the  nature  of 
the  task. 

5.  Conclusions 


In  general,  participants  felt  the  MMI  was  fairly  easy  to  use,  was  simple  and  straight 
forward,  and  could  greatly  benefit  Soldiers  in  terms  of  mission  safety  by  supporting 
remote  interaction  with  a  robot  teammate.  As  expected,  concerns  the  visual  display 
could  distract  or  allocate  the  attentional  resources  of  the  Soldiers  away  from  their 
primary  tasking  were  expressed  in  the  free-response  questionnaire.  This  is  a 
primary  reason  the  US  Army  is  developing  visual  displays  on  mobile  platforms 
(Young  2014);  the  intention  is  these  displays  will  only  be  used  for  quick  reference 
or  response  and  should  not  require  extensive  viewing  time.  Additionally,  part  of  the 
US  Army’s  Robotics  Collaborative  Technology  Alliance  (RCTA  2014)  research 
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interests  focuses  on  the  use  of  other  modalities  that  do  not  require  a  visual  display 
for  bidirectional  communication  between  dismounted  Soldier-robot  teammates. 

In  addition,  reporting  frequency  was  impacted  by  primary  task  difficulty,  which 
supports  the  idea  of  adaptive  automation  that  manages  the  frequency  of 
robot-to-human  information  sharing.  While  more  participants  showed  a  preference 
for  reports  that  condensed  and  summarized  information  (i.e.,  reports  displayed  in  a 
text  format),  participants  still  liked  having  both  report  formats  available.  For  future 
MMI  display  designs  this  may  mean  redundancy  in  the  presentation  of  information 
will  likely  be  considered  valuable  to  users. 
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Demographics  Questionnaire 


1.  General  Information 

a.  Age  (yrs): _  b.  Gender:  _ M  _ F 

c.  Do  you  have  corrected  vision?  _ None  _ Glasses  _ Contact  Lenses 

d.  Do  you  have  any  type  of  color  blindness/color  vision  deficiency? _ Yes  _ No 

e.  What  is  your  native  language?  _ _ 

f.  Do  you  speak  more  than  one  language?  _ Yes  _ No 

g.  If  you  answered  YES  to  question  l.f,  how  fluent  would  you  rate  your  ability  to 
speak  a  secondary  language? 


1 

2 

3 

4 

5 

6 

Very  low 

Low 

Moderate 

Above 

High 

Very  high 

fluency 

fluency 

fluency 

moderate 

fluency 

fluency 

fluency 


2.  Military  Experience 

a.  Flow  many  years  have  you  been  in  the  military? _  b.  Current  rank _ 

c.  What  is  your  MOS? _ 

d.  Please  list  all  combat  deployments  (Iraq,  Afghanistan,  etc.)  and  the  length  (Years  / 
Months)  of  each. 

Location  Time 


e.  Do  you  have  operational  experience  in  complex  urban  terrain? _ Yes _ No 

f.  Do  you  have  operational  experience  in  reconnaissance  situations? _ Yes _ No 
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3.  Education 


a.  What  is  your  highest  level  of  education  received?  Select  one. 

_ GED 

_ High  School 

_ Some  College 

_ Bachelor’s  Degree 

_ M.S/M.A 

_ Ph.D  or  other  doctorate. 

Other: _ 

b.  If  applicable,  what  subject  is  your  degree  in  (for  example,  Criminal  justice)? 


4.  Computer  Experience 


a.  How  long  have  you  been  using  a  computer? 


1 

2 

3 

4 

5 

6 

Never 

Less  than 

1  year 

1-3  years 

4-6  years 

7-10  years 

10  years  or 
more 

b.  How  often  do  you  use  ; 

a  computer? 

1 

2 

3 

4 

5 

6 

Less  than 

1  hour  a 
day 

1-2  hours  a 
day 

Over  2 
hours  a 
day 

Weekly 

Monthly 

A  few 
times  a 
year 

c.  For  each  of  the  following  questions,  circle  the  response  that  best  describes  how 

often  you: 

Use  a  mouse: 

1 

2 

3 

4 

5 

6 

Never  Rarely 

Use  a  joystick: 

Once  every 
few 

months 

Monthly 

Weekly 

Daily 

1 

2 

3 

4 

5 

6 

Never  Rarely 

Use  a  touch  screen: 

Once  every 
few 

months 

Monthly 

Weekly 

Daily 

1 

2 

3 

4 

5 

6 

Never 

Rarely 

Once  every 

Monthly 

Weekly 

Daily 

few 

months 


Approved  for  public  release;  distribution  is  unlimited. 


47 


Use  icon-based  programs/software: 


12  3  4 

5 

6 

Never  Rarely  Once  every  Monthly 

few 

months 

Use  programs/software  with  pull-down  menus: 

Weekly 

Daily 

12  3  4 

5 

6 

Never  Rarely  Once  every  Monthly 

few 

months 

Use  a  graphics/drawing  features  in  software  packages: 

Weekly 

Daily 

12  3  4 

5 

6 

Never  Rarely  Once  every  Monthly 

few 

months 

5.  Video  Game  Experience 

a.  Please  indicate  how  often  you  play  video  games: 

Weekly 

Daily 

12  3  4 

5 

6 

Never  Rarely  Once  every  Monthly 

few 

months 

Weekly 

Daily 

b.  Please  indicate  how  you  would  rate  your  experience  in 
video  games: 

working  with  any  type  of 

12  3  4 

5 

6 

Not  at  all  Somewhat  Moderately  Above 

Highly 

Very 

familiar  familiar  familiar  moderately 

familiar 

familiar 

highly 

familiar 

c.  Which  type  of  video  game  do  you  play  most  often? 

_ Action-adventure  _ Serious  games/Educational 

_ First  person  shooters  _ Simulation 

_ Military-based  _ Strategy 

_ Mobile/cellphone  games  _ Sports 

_ Multiplayer  online  gaming  _ Other,  please  indicate  which  one: 

_ Role  playing  _ 


6.  Robotics  Experience 

a.  Have  you  any  experience  with  military  robots?  _ Yes _ No 

b.  If  you  answered  YES  to  question  6.a,  what  type  of  robots  and  for  what  purpose? 

Type  Purpose 
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c.  Please  indicate  how  you  would  rate  your  level  of  experience  with  any  robots: 


1 

2 

3 

4 

5 

6 

Not  at  all 

Somewhat 

Moderately 

Above 

Highly 

Very 

familiar 

familiar 

familiar 

moderately 

familiar 

highly 

familiar 

familiar 

d.  Please  indicate  how  you  would  rate  your  level  of  knowledge  regarding  robotics 

technology  (e.g.,  pack  bot,  big  dog,  talon,  AIBO  etc.): 

1 

2 

3 

4 

5 

6 

Not  at  all 

Somewhat 

Moderately 

Above 

Highly 

Very 

familiar 

familiar 

familiar 

moderately 

familiar 

highly 

familiar 

familiar 
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Intentionally  lelt  blank. 
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Appendix  B.  Cube  Comparison  Questionnaire 
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Cube  Comparison  Test 


Name 


CUBE  COMPARISONS  TEST  —  S-2  (Rev.) 

Wooden  blocks  such  as  children  play  with  are  often  cubical  with  a  different 
letter,  number,  or  symbol  on  each  of  the  six  faces  (top,  bottom,  four  sides). 

Each  problem  in  this  test  consists  of  drawings  of  pairs  of  cubes  or  blocks  of 
this  kind.  Remember,  there  is  a  different  design,  number,  or  letter  on  each  face 
of  a  given  cube  or  block.  Compare  the  two  cubes  in  each  pair  below. 


S  CD  D  CBS  StKI  DO 


The  first  pair  is  marked  D  because  they  must  be  drawings  of  different  cubes. 

If  the  left  cube  is  turned  so  that  the  A  is  upright  and  facing  you,  the  N  would  be 
to  the  left  of  the  A  and  hidden,  not  to  the  right  of  the  A  as  is  shown  on  the  right 
hand  member  of  the  pair.  Thus,  the  drawings  must  be  of  different  cubes. 

The  second  pair  is  marked  S  because  they  could  be  drawings  of  the  same  cube. 
That  is,  if  the  A  is  turned  on  its  side  the  X  becomes  hidden,  the  B  is  now  on  top, 
and  the  C  (which  was  hidden)  now  appears.  Thus  the  two  drawings  could  be  of  the 
same  cube. 

Note :  No  letters,  numbers,  or  symbols  appear  on  more  than  one  face  of  a  given 

cube.  Except  for  that,  any  letter,  number  or  symbol  can  be  on  the  hidden  faces  of 
a  cube. 


Work  the  three  examples  below. 


S  CD  D  CD  S  CZ>  DO  SCO  0  CD 


The  first  pair  immediately  above  should  be  marked  D  because  the  X  cannot  be  at 
the  peak  of  the  A  on  the  left  hand  drawing  and  at  the  base  of  the  A  on  the  right 
hand  drawing.  The  second  pair  is  "different"  because  P  has  its  side  next  to  G  on 
the  left  hand  cube  but  its  top  next  to  G  on  the  right  hand  cube.  The  blocks  in  the 
third  pair  are  the  same,  the  J  and  K  are  just  turned  on  their  side,  moving  the  0  to 
the  top. 

Your  score  on  this  test  will  be  the  number  marked  correctly  minus  the  number 
marked  incorrectly.  Therefore,  it  will  not  be  to  your  advantage  to  guess  unless  you 
have  some  idea  which  choice  is  correct.  Work  as  quickly  as  you  can  without  sacri¬ 
ficing  accuracy. 

You  will  have  3  minutes  for  each  of  the  two  parts  of  this  test.  Each  part  has 
one  page.  When  you  have  finished  Part  1,  STOP. 

DO  NOT  TURN  THE  PAGE  UNTIL  YOU  ARE  ASKED  TO  DO  SO. 
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Page  2 

Part  1  ( 3  minutes  ) 
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DO  NOT  GO  ON  TO  THE  NEXT  PAGE  UNTIL  ASKED  TO  DO  SO.  STOP. 
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Intentionally  left  blank. 
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Appendix  C.  NASA-Task  Load  Index  (NASA-TLX) 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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NASA-TLX  Questionnaire 


Please  rate  your  overall  impression  of  demands  imposed  on  you  during  the  exercise. 


1.  Mental  Demand:  How  much  mental  and  perceptual  activity  was  required  (e.g., 
thinking,  looking,  searching,  etc.)?  Was  the  task  easy  or  demanding,  simple  or 
complex,  exacting  or  forgiving? 

LOW  I— -I— -I— -I— -I— -I— -I— -I— -I— -I  HIGH 
1  2  345678  9  10 

2.  Physical  Demand:  How  much  physical  activity  was  required  (e.g.,  pushing, 
pulling,  turning,  controlling,  activating,  etc.)?  Was  the  task  easy  or  demanding, 
slow  or  brisk,  slack  or  strenuous,  restful  or  laborious? 

LOW  I— -I— -I— -I— -I— -I— -I— -I— -I— -|  HIGH 
1  2  345678  9  10 

3.  Temporal  Demand:  How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace 
at  which  the  task  or  task  elements  occurred?  Was  the  pace  slow  and  leisurely  or 
rapid  and  frantic? 

LOW  I— -I— -I— -I— -I— -I— -|— -I— -I— -|  HIGH 
1  2  345678  9  10 

4.  Level  of  Effort:  How  hard  did  you  have  to  work  (mentally  and  physically)  to 
accomplish  your  level  of  performance? 

LOW  I— -I— - 1 — I— -I— -I— -I— - 1  HIGH 
1  2  345678  9  10 

5.  Level  of  Frustration:  How  insecure,  discouraged,  irritated,  stressed  and  annoyed 
versus  secure,  gratified,  content,  relaxed  and  complacent  did  you  feel  during  the 
task? 

LOW  |- — |— -|— -I— -|— -I — I— -I— -I— -I  HIGH 
1  2  345678  9  10 

6.  Performance:  How  successful  do  you  think  you  were  in  accomplishing  the  goals 
of  the  task  set  by  the  experimenter  (or  yourself)?  How  satisfied  were  you  with  your 
performance  in  accomplishing  these  goals? 

GOOD  I— -I— -I— -I— -I— -I— -I— -I— - 1— -I  POOR 
1  2  345678  9  10 
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Appendix  D.  System  Usability  Scale 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Comparison  of  System  Usability  Questionnaire 


Please  circle  the  response  you  feel  is  most  accurate 

1. 1  think  that  I  would  like  to  use  the  device  frequently 

Strongly 
disagree 


Strongly 

agree 


2. 1  found  the  device  unnecessarily  complex 

Strongly 
disagree 


Strongly 

agree 


3. 1  thought  the  device  was  easy  to  use 

Strongly 
disagree 


Strongly 

agree 


4. 1  think  that  I  would  need  the  support  of  a  technical  person  to  be  able  to  use 
this  device 


Strongly 

disagree 


Strongly 

agree 


5. 1  found  the  various  functions  in  this  device  were  well  integrated 


Strongly 

disagree 


Strongly 

agree 
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6. 1  thought  there  was  too  much  inconsistency  in  this  device 


Strongly 

disagree 


Strongly 

agree 


7. 1  would  imagine  that  the  most  people  would  learn  to  use  this  device  very 
quickly 


Strongly 

disagree 


Strongly 

agree 


8. 1  found  the  device  very  cumbersome  to  use 

Strongly 
disagree 


Strongly 

agree 


9. 1  felt  very  confident  using  the  device 

Strongly 
disagree 


Strongly 

agree 


10. 1  needed  to  learn  a  lot  of  things  before  I  could  get  going  with  this  device 


Strongly 

disagree 


Strongly 

agree 
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Appendix  E.  Free  Response  Questionnaire 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Free  Response  Questionnaire 


Instructions:  Please  answer  the  following  questions  about  your  experience  with 
the  device. 

1.  Please  list  the  most  positive  aspect(s)  of  the  device: 


2.  Please  list  the  most  negative  aspect(s)  of  the  device: 


3.  What  do  you  think  of  the  way  in  which  the  device  provided  building 
identification  information?  Did  you  have  a  preference? 


4.  What  do  you  think  of  the  way  in  which  the  device  provided  navigation 
information?  Did  you  have  a  preference? 


5.  What  else  did  you  think  was  important,  remarkable  or  surprising  about  your 
interaction  with  the  robot  using  the  device?  Additional  comments: 


6.  Please  comment  on  any  features  of  the  device  that  you  think  had  an  impact  on 
your  interactions  with  the  robot.  You  can  comment  on  both  positive  and  negative 
features,  and  suggest  improvements. 
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Appendix  F.  Situation  Awareness  Global  Assessment  Technique 

(SAGAT)  Probes 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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SAGAT  Probes 


SA-1  Probes 

“1.  Based  on  the  currently  available  robot  report,  is  the  robot’s  immediate 
environment  safe?” 

“2.  Up  to  this  point,  how  many  critical  threats  have  been  seen  on  the  Soldier’ 
route?” 

SA-2  Probes 

“1.  Was  the  robot’s  route  safe?” 

“2.  Was  the  Soldier’s  route  safe?” 
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Appendix  G.  Situation  Awareness  Rating  Technique  (SART), 

Adapted 
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SART  (Adapted) 


Please  rate  your  awareness  of  the  situation  by  circling  a  number,  according  to 
the  following  dimensions. 

Dimension  1:  Demand  on  attentional  resources 


Low  High 

Instability: 

How  likely  did  you  feel  that  the 
demands  of  this  mission  and 
anything  related  to  the  situation 
would  suddenly  change? 

1 

2 

3 

4 

5 

6 

7 

Complexity: 

How  complex  did  you  feel  this 
mission  and  anything  related  to 
the  situation  was? 

1 

2 

3 

4 

5 

6 

7 

Variability: 

How  many  elements  of  the 
mission  did  you  feel  were 
changing  in  this  mission  and 
anything  related  to  the  situation? 

1 

2 

3 

4 

5 

6 

7 

Dimension  2:  Supply  of  attentional  resources 


Low  High 

Arousal: 

How  ready  did  you  feel  you  were 
in  completing  this  mission  and 
anything  related  to  the  situation? 

1 

2 

3 

4 

5 

6 

7 

Spare  Mental  capacity: 

How  much  attention  did  you  feel 
you  had  left  and  could  direct  to 
other  tasks? 

1 

2 

3 

4 

5 

6 

7 

Concentration: 

Did  you  feel  you  were  able  to 
concentrate  on  completing  this 
mission  and  anything  related  to 
the  situation? 

1 

2 

3 

4 

5 

6 

7 

Division  of  attention: 

How  much  attention  did  you 
direct  towards  this  mission  and 
anything  related  to  the  situation? 

1 

2 

3 

4 

5 

6 

7 
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Dimension  3:  Your  understanding  of  the  situation 


Low  High 

Information  quantity: 

How  much  information  did  you 
feel  you  understood  while 
completing  the  mission? 

1 

2 

3 

4 

5 

6 

7 

Information  quality: 

How  good  was  the  information 
you  received  while  completing 
this  mission? 

1 

2 

3 

4 

5 

6 

7 

Familiarity: 

How  much  knowledge  did  you 
feel  you  had  about  this  mission 
and  anything  related  to  the 
situation? 

1 

2 

3 

4 

5 

6 

7 

Please  rate  your  awareness  of  the  robot’s  behaviors  by  circling  a  number, 
according  to  the  following  dimensions. 

Dimension  4:  Situation  awareness  of  the  robot 


Low 

High 

Perception: 

Please  rate  your  knowledge  of  the 
robot’s  location,  its  surrounding 
environment,  and  its  status  during 
the  course  of  the  mission. 

1 

2 

3 

4 

5 

6 

7 

Comprehension: 

Please  rate  your  ability  to  derive 
meaning  from  perception  of  the 
robot  (e.g.,  its  location, 
surroundings,  and  status)  while 
completing  the  mission. 

1 

2 

3 

4 

5 

6 

7 

Projection: 

Please  rate  your  ability  to  predict 
how  the  robot  would  behave  in 
the  near  future,  while  completing 
the  mission. 

1 

2 

3 

4 

5 

6 

7 
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Appendix  H.  Cheat  Sheet  for  Robot-Reporting  Scenarios 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Rules  to  Remember 


Safety  of  robot's  environment 

Safety  of  robot's  Route 

3  or  more  threats 


Critical  threat 


3  or  more  critical  threats 
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Appendix  I.  Informed  Consent 
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Informed  Consent  to  Participate  in  Research  Form 
Army  Research  Laboratory 
Human  Research  &  F.ngineering  Directorate 
Aberdeen  Proving  Ground,  MD  21005 

Project  Title:  Squad  Level  Soldier-Robot  Communication  Exchanges 

Project  Number:  ARL- 

Principal  Investigators:  Dr.  Linda  Elliott 

Human  Research  and  Engineering  Directorate 
Human  Factors  Integration  Div  ision 
Weapons  Branch.  MCOE  Field  Element 
(706)  545-5634,  linda.r.clliott.civ  u  mail.mil 

Dr.  Daniel  Barber 
University  of  Central  Florida 
Institute  for  Simulation  and  Training 
(407)  882-1 128,  dbarber  a  ist.ue f.cdu 


ARL 


You  are  being  asked  to  participate  in  a  simulation-based  assessment  into  differences  in 
expectations  of  robot  information  sharing  and  information  requesting.  This  investigation  will 
focus  on  your  requests  for  information  from  a  robot  in  the  form  of  status  updates  and  preferences 
for  how  robots  should  request  assistance.  This  consent  form  explains  the  evaluation  and  your 
part  in  it.  Please  read  this  form  carefully  before  you  decide  to  take  part.  You  can  take  as  much 
time  as  you  need.  Please  ask  any  questions  at  any  time  about  anything  you  do  not  understand. 
You  are  a  volunteer.  At  any  moment,  you  may  withdraw  from  the  experiment  without 
consequences. 
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Purpose  of  the  Study 

The  purpose  for  this  study  is  to  collect  information  regarding  how  future  robot  teammates  should 
communicate  with  Soldiers  during  reconnaissance  and  surveillance  activities.  Specifically,  we 
are  interested  in  when  a  robot  should  report  information,  what  type  of  information  is  needed, 
how  navigational  assistance  requests  from  a  robot  are  presented,  and  how  a  robot  should  move 
when  given  specific  commands.  Your  participation  will  provide  additional  understanding  into 
how  future  robots  should  share  and  request  information  with  Soldiers  while  they  perform 
reconnaissance  tasks.  Results  from  this  effort  will  drive  requirements  for  future  Soldier-Robot 
teaming  concepts  currently  under  development. 

Test  Procedures 

You  will  be  asked  to  complete  a  demographics  questionnaire  concerning  your  military 
background  and  experience.  You  will  also  be  asked  to  complete  a  3-minute  spatial  ability  test. 
Next  you  will  be  trained  how  to  perform  the  tasks  within  the  simulation  environment  used  for 
this  study.  First  you  will  be  trained  on  the  different  types  of  targets  within  the  simulation, 
differentiating  threats  and  non-threats.  You  will  then  perform  a  practice  task  where  you  will  be 
required  to  classify  targets  (i.e.  people,  objects)  considered  threats  based  on  the  previous 
training.  Upon  completion  of  this  practice  task,  you  will  then  receive  training  on  when  and  how 
to  request  information  from  the  simulated  robot.  A  practice  task  will  be  provided  to  familiarize 
you  with  the  task  and  what  information  you  must  remember.  Next,  you  will  receive  training  and 
practice  on  w  hat  type  of  navigation  assistance  requests  the  robot  will  ask,  and  how  to  respond  to 
them.  After  completing  training  on  the  individual  tasks  you  will  be  given  two  additional  practice 
scenarios:  one  with  threat  detection  and  robot  information  requests,  and  one  with  threat  detection 
and  robot  assistance  requests. 

After  completing  training  on  how  to  perform  the  tasks,  you  will  then  execute  three  scenarios 
applying  what  was  learned  in  the  training.  In  two  of  the  scenarios,  you  will  perform  a  threat 
detection  task  while  requesting  information  from  a  simulated  robot  performing  a  reconnaissance 
task.  In  the  third  scenario  you  will  perform  the  same  threat  detection  task  while  answering 
navigation  assistance  requests  from  a  simulated  robot  performing  reconnaissance.  For  all 
scenarios  we  will  be  recording  when  you  communicate  with  the  robot,  the  type  of  information 
requested,  and  models  classified  during  the  threat  detection  task.  After  completing  each  of  the 
three  scenarios  you  will  be  asked  to  complete  the  NASA-TLX  and  SART  questionnaires  to 
measure  your  workload  and  situation  awareness  respectively.  Finally,  after  you  have  completed 
all  scenarios,  you  will  be  asked  to  complete  a  "Robot  Movement  Questionnaire.”  This 
questionnaire  will  provide  additional  insight  into  your  expectations  of  how'  a  robot  should  move 
under  different  circumstances. 
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Discomforts  and  Risks 


This  study  should  offer  minimal  risks  to  your  health  and  well-being.  You  can  choose  to 
withdraw  from  the  experiment  at  any  time,  or  to  take  a  rest  break  at  any  time. 

Benefits 

You  will  receive  no  benefits  from  participating  in  the  experiment,  other  than  the  personal 
satisfaction  of  supporting  the  Army's  research  in  developing  improvements  in  Soldier 
equipment. 

Duration 

Your  participation  in  this  experiment  will  take  approximately  2  hours. 

Confidentiality 

Your  participation  in  this  research  is  confidential.  The  data  will  be  stored  and  secured  in  the 
offices  of  the  principal  investigator  in  a  locked  file  cabinet.  The  data,  without  any  identifying 
information,  will  be  transferred  to  a  password-protected  computer  for  data  analysis.  This  consent 
form  will  be  retained  by  the  principal  investigator  for  a  minimum  of  three  years. 

If  the  results  of  the  experiment  are  published  or  presented  to  anyone,  no  personally  identifiable 
information  will  be  shared.  Publication  of  the  results  of  this  study  in  a  journal  or  technical 
report,  or  presentation  at  a  meeting,  will  not  reveal  personally  identifiable  information.  The 
research  staff  will  protect  your  data  from  disclosure  to  people  not  connected  with  the  study. 
However,  complete  confidentiality  cannot  be  guaranteed  because  officials  of  the  U.  S.  Army 
Human  Research  Protections  Office  and  the  Army  Research  Laboratory’s  Institutional  Review 
Board  are  permitted  by  law  to  inspect  the  records  obtained  in  this  study  to  insure  compliance 
with  laws  and  regulations  covering  experiments  using  human  subjects. 

We  would  like  your  permission  to  take  pictures/video  during  the  experimental  session.  The 
pictures  will  be  printed  in  technical  reports  and  shown  during  presentations  when  we  describe  the 
results  of  the  study.  To  protect  your  identity,  we  will  ask  you  to  remove  your  name  badge  and 
we  will  pixelate  the  image  to  obscure  your  face.  You  can  still  be  in  the  study  if  you  prefer  not  to 
be  photographed/videotaped.  Please  indicate  below  if  you  will  agree  to  allow  us  to  take  pictures 
of  you. 

I  give  consent  to  be  photographed/video  during  this  study:  _ Y es  _ No 

please  initial: _ 


Contact  Information  for  Additional  Questions 

You  have  the  right  to  obtain  answers  to  any  questions  you  might  have  about  this  research  at 
anytime  during  this  test.  Please  contact  anyone  listed  at  the  top  of  the  first  page  of  this  consent 
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form  for  more  information  about  this  study.  You  may  also  contact  the  Chairperson  of  the  Army 
Research  Lab  Institutional  Review  Board,  at  (4 1 0)  278-5992  or  (DSN)  298-5992  with  questions, 
complaints,  or  concerns  about  this  research,  or  if  you  feel  this  study  has  harmed  you.  The 
Chairperson  can  also  answer  questions  about  your  rights  as  a  research  participant.  You  may  also 
call  this  number  if  you  cannot  reach  the  research  team  or  wish  to  talk  to  someone  else. 

Voluntary  Participation 

Your  decision  to  be  in  this  evaluation  is  voluntary.  You  can  stop  at  any  time.  You  do  not  have  to 
answer  any  questions  you  do  not  want  to  answer.  Refusal  to  take  part  in  or  withdrawal  from  this 
study  will  involve  no  penalty  or  loss  of  benefits  you  would  receive  by  staying  in  it.  Military 
personnel  cannot  be  punished  under  the  Uniform  Code  of  Military  Justice  for  choosing  not  to 
take  part  in  or  withdrawing  from  this  study,  and  cannot  receive  administrative  sanctions  for 
choosing  not  to  participate.  Civilian  employees  or  contractors  cannot  receive  administrative 
sanctions  for  choosing  not  to  participate  in  or  withdrawing  from  this  study.  You  must  be  18  years 
of  age  or  older  to  take  part  in  this  research  study.  If  you  agree  to  take  part  in  this  research  study- 
based  on  the  information  outlined  above,  please  sign  your  name  and  indicate  the  date  below.  You 
will  be  given  a  copy  of  this  consent  form  for  your  records. 

This  consent  form  is  approved  from  XXX  to  XXX. 


Participant's  Signature 

Date 

Participant’s  Printed  Name 

Signature  of  Person  Obtaining  Consent 

Date 

Primed  Name  of  Person  Obtaining  Consent 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ANOVA 

B 

D 

DARPA 

G 

HRI 

M 

MIX 

MMC 

MMI 

NASA 

P 

POV 

R 

RA 

RR 

SA 

SAGAT 

SART 

SD 

SUS 

TD 

TLX 

XUV 


analysis  of  variance 
blue 

Cohen’s  d  statistical  value 

Defense  Advanced  Research  Projects  Agency 

green 

human-robot  interface 
mean 

Mixed  Initiative  experimental 
multimodal  communication 
multimodal  interface 

National  Aeronautics  and  Space  Administration 
probability  of  rejection  of  the  hypothesis 
point  of  view 
red 

robot  assistance 
robot  reporting 
situation  awareness 

Situation  Awareness  Global  Assessment  Technique 

Situation  Awareness  Rating  Technique 

standard  deviation 

System  Usability  Scale 

threat  detection 

Task  Load  Index 

experimental  unmanned  vehicle 
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